Nov 04 02:31:48.710868 ip-10-0-133-209 systemd[1]: Starting Kubernetes Kubelet... Nov 04 02:31:49.098935 ip-10-0-133-209 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:49.098935 ip-10-0-133-209 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 02:31:49.098935 ip-10-0-133-209 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:49.098935 ip-10-0-133-209 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:49.103066 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.102980 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.103066 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103066 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.103066 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103070 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103074 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103077 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103080 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103083 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103086 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103088 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103093 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103095 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103098 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103101 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103103 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103106 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103110 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103112 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103115 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103123 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103126 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103129 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103132 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.103162 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103134 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103137 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103140 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103143 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103145 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103148 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103151 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103153 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103156 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103159 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103161 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103164 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103167 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103170 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103172 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103175 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103178 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103181 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103183 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103186 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.103637 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103188 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103191 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103193 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103197 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103200 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103203 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103205 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103207 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103210 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103212 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103215 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103217 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103220 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103223 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103226 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103229 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103231 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103234 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103236 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103239 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.104099 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103241 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103244 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103246 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103250 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103252 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103255 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103257 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103260 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103262 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103265 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103267 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103270 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103289 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103296 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103302 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103306 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103311 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103314 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103316 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103319 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.104604 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103322 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103324 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103327 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103330 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103332 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103335 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103338 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103341 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103344 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103346 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103349 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103352 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103354 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103357 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103362 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103366 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103368 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103372 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103374 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.105063 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103377 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103380 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103382 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103385 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.103387 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104243 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104251 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104254 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104257 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104260 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104269 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104288 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104291 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104293 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104296 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104300 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104302 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104305 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104308 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.105534 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104311 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104315 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104319 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104323 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104326 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104329 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104332 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104334 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104337 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104339 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104342 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104345 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104347 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104349 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104352 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104354 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104357 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104359 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104362 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.106026 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104364 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104367 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104369 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104372 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104374 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104377 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104379 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104383 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104385 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104388 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104390 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104393 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104395 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104398 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104401 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104404 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104406 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104409 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104411 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.106480 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104414 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104416 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104419 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104422 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104424 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104426 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104429 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104434 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104436 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104439 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104441 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104444 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104446 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104448 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104451 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104456 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104460 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104462 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104466 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.106929 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104468 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104471 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104473 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104476 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104478 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104481 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104484 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104487 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104489 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104492 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104495 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104498 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104500 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104502 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104505 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104507 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104510 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104513 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104515 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104517 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.107436 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104520 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104522 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104525 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104527 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104530 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104532 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104535 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104537 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104540 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104542 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104545 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104548 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104552 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.104554 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105098 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105103 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105107 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105110 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105113 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105116 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.107913 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105118 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105121 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105123 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105129 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105131 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105134 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105136 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105139 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105141 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105144 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105146 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105149 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105151 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105154 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105156 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105159 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105161 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105164 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105166 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.108402 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105172 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105175 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105177 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105180 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105183 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105186 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105188 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105191 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105193 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105196 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105201 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105203 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105206 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105209 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105211 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105214 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105216 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105219 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105221 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.108867 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105224 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105226 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105229 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105231 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105234 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105236 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105239 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105241 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105244 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105246 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105249 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105252 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105254 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105258 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105261 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105263 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105266 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105268 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105287 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.109309 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105291 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105293 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105296 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105298 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105302 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105305 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105308 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105311 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105314 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105319 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105322 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105324 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105327 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105329 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105332 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105334 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105337 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105340 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105342 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105345 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.109756 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105347 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105350 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105352 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105355 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105357 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105360 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105363 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105366 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105369 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105371 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105373 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105376 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105381 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105384 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105387 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105389 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105393 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105397 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105400 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105403 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.110232 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105406 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.105410 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106668 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106678 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106684 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106690 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106694 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106698 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106703 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106707 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106710 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106713 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106716 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106720 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106723 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106726 2575 flags.go:64] FLAG: --cgroup-root="" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106729 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106732 2575 flags.go:64] FLAG: --client-ca-file="" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106738 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106741 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106747 2575 flags.go:64] FLAG: --cluster-domain="" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106750 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106753 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106756 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 02:31:49.110711 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106759 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106764 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106767 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106770 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106775 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106778 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106781 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106784 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106787 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106790 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106795 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106798 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106801 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106804 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106807 2575 flags.go:64] FLAG: --enable-server="true" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106810 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106815 2575 flags.go:64] FLAG: --event-burst="100" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106818 2575 flags.go:64] FLAG: --event-qps="50" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106821 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106824 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106827 2575 flags.go:64] FLAG: --eviction-hard="" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106831 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106834 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106837 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106840 2575 flags.go:64] FLAG: --eviction-soft="" Nov 04 02:31:49.111271 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106843 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106848 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106851 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106854 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106857 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106860 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106863 2575 flags.go:64] FLAG: --feature-gates="" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106868 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106871 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106875 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106878 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106882 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106885 2575 flags.go:64] FLAG: --help="false" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106888 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106891 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106894 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106897 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106901 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106904 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106907 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106910 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106913 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106916 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 02:31:49.111923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106919 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106922 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106925 2575 flags.go:64] FLAG: --kube-reserved="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106928 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106932 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106935 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106939 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106942 2575 flags.go:64] FLAG: --lock-file="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106945 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106948 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106952 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106958 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106961 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106964 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106967 2575 flags.go:64] FLAG: --logging-format="text" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106970 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106973 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106976 2575 flags.go:64] FLAG: --manifest-url="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106979 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106984 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106988 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106992 2575 flags.go:64] FLAG: --max-pods="110" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106995 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.106998 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107001 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 02:31:49.112561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107004 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107007 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107010 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107013 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107020 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107023 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107026 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107029 2575 flags.go:64] FLAG: --pod-cidr="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107031 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107034 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107038 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107041 2575 flags.go:64] FLAG: --port="10250" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107044 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107046 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d2b6b6e3a25a8564" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107050 2575 flags.go:64] FLAG: --qos-reserved="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107053 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107056 2575 flags.go:64] FLAG: --register-node="true" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107059 2575 flags.go:64] FLAG: --register-with-taints="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107063 2575 flags.go:64] FLAG: --registry-burst="10" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107066 2575 flags.go:64] FLAG: --registry-qps="5" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107068 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107071 2575 flags.go:64] FLAG: --reserved-memory="" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107075 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107078 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107081 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107084 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 02:31:49.113155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107088 2575 flags.go:64] FLAG: --runonce="false" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107091 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107095 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107098 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107101 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107104 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107107 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107110 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107113 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107116 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107119 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107122 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107125 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107128 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107131 2575 flags.go:64] FLAG: --system-cgroups="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107133 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107139 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107142 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107145 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107149 2575 flags.go:64] FLAG: --tls-min-version="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107152 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107155 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107157 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107160 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107164 2575 flags.go:64] FLAG: --v="2" Nov 04 02:31:49.114191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107168 2575 flags.go:64] FLAG: --version="false" Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107172 2575 flags.go:64] FLAG: --vmodule="" Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107175 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107179 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107350 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107359 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107362 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107364 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107368 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107372 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107374 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107377 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107380 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107383 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107385 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107387 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107390 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107393 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107395 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107398 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.115270 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107400 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107403 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107405 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107407 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107410 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107413 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107415 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107418 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107420 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107423 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107425 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107428 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107430 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107433 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107435 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107438 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107440 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107443 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107446 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.116136 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107458 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107462 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107465 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107468 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107471 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107474 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107476 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107479 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107481 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107484 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107486 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107489 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107491 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107494 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107496 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107501 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107504 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107508 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107512 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.116728 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107515 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107517 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107523 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107528 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107530 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107533 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107535 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107538 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107540 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107543 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107546 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107548 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107552 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107555 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107558 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107561 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107564 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107566 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.117366 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107569 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107571 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107574 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107576 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107579 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107581 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107584 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107586 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107590 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107592 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107595 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107597 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107600 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107603 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107605 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107608 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107610 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107613 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107615 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107618 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.118371 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107620 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107623 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107625 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107628 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107631 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107635 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107639 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107642 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107645 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107648 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107651 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107653 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.107707 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107745 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107749 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107752 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.119250 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107755 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107758 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107761 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107763 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107766 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107770 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107773 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107775 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107778 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107780 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107783 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107786 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107788 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107791 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107793 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107796 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107798 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107801 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107803 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.119730 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107806 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107809 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107813 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107817 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107820 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107823 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107825 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107828 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107831 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107833 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107836 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107838 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107841 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107843 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107846 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107848 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107851 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107853 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107857 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.120508 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107860 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107862 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107865 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107868 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107870 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107873 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107875 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107878 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107881 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107883 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107886 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107888 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107891 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107893 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107896 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107898 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107902 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107904 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107907 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107909 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.121297 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107912 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107914 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107918 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107921 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107923 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107926 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107929 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107931 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107935 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107938 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107942 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107946 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107949 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107952 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107954 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107957 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107960 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107962 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107965 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.121981 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107967 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107971 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107973 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107975 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107978 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107980 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107983 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107985 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107988 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107991 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107994 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107996 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.107999 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108001 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108004 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108006 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108009 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108013 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108015 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.122610 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108017 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108020 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108022 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108025 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108027 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108031 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.108034 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.116350 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.116373 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.116994 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.117005 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117112 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117121 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117128 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117132 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117136 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117141 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117145 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117150 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117154 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.123070 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117159 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117163 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117167 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117171 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117175 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117179 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117184 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117188 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117192 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117197 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117201 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117206 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117210 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117214 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117218 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117222 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117227 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117232 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117236 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.123625 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117240 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117256 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117261 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117265 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117269 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117295 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117300 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117308 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117313 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117317 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117321 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117327 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117333 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117337 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117342 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117346 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117350 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117355 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.124060 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117359 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117363 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117367 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117371 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117375 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117379 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117383 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117388 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117392 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117396 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117400 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117404 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117409 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117414 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117419 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117432 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117436 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117440 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117445 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.124545 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117450 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117456 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117463 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117467 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117472 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117476 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117480 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117484 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117488 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117493 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117497 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117502 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117506 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117510 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117514 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117518 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117522 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117535 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117539 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.125043 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117543 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117547 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117551 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117556 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117560 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117565 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117569 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117573 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117578 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117582 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117594 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117599 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117603 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117607 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117612 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117616 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117620 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117625 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117629 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117633 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.125522 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117637 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.117681 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117980 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117991 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.117997 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118002 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118006 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118011 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118017 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118022 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118026 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118031 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118035 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118039 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118043 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.126215 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118048 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118052 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118057 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118061 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118065 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118071 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118075 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118079 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118098 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118102 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118107 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118111 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118115 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118119 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118124 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118128 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118132 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118137 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118141 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.126903 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118145 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118149 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118153 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118158 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118162 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118167 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118171 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118175 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118179 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118183 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118187 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118191 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118196 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118200 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118204 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118209 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118213 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118218 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118222 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118226 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.127415 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118230 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118234 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118249 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118254 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118258 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118263 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118267 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118292 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118299 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118315 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118321 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118325 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118329 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118334 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118337 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118341 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118345 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118349 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118353 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.127998 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118357 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118361 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118365 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118369 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118373 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118377 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118381 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118385 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118389 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118393 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118397 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118401 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118406 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118410 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118414 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118418 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118436 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118440 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118444 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.128469 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118449 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118453 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118457 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118461 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118466 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118470 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118474 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118478 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118483 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118487 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118491 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118495 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118500 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118504 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118508 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.118553 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:49.128947 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118615 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118623 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118627 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118632 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118636 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118641 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118645 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118649 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118654 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118658 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118662 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118666 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118673 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118688 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118694 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118699 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118704 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118708 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118713 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118717 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:49.129340 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118721 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118726 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118730 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118734 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118738 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118743 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118747 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118751 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118755 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118759 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118764 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118768 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118772 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118777 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118781 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118785 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118789 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118794 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118798 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118802 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:49.129814 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118807 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118812 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118816 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118820 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118824 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118829 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118841 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118845 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118849 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118853 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118858 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118862 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118867 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118871 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118875 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118880 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118884 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118888 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118893 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:49.130291 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118899 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118905 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118911 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118916 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118920 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118925 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118929 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118933 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118937 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118942 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118946 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118950 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118954 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118962 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118966 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118970 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118975 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118979 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118983 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:49.130745 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.118987 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119001 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119005 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119009 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119013 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119017 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119021 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119026 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119030 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119034 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119039 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119043 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119047 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119051 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119055 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119069 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119073 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119078 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119082 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:49.131179 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119086 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119091 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119095 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119099 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119103 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119107 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119111 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119117 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:49.119124 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.119226 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.122230 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.123433 2575 server.go:1014] "Starting client certificate rotation" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.123537 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.126982 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.128775 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:31:49.131662 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.131474 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:31:49.143030 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.143010 2575 log.go:25] "Validated CRI v1 runtime API" Nov 04 02:31:49.146588 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.146569 2575 log.go:25] "Validated CRI v1 image API" Nov 04 02:31:49.148046 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.148015 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 02:31:49.151956 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.151929 2575 fs.go:135] Filesystem UUIDs: map[43e7832b-81cf-49b6-b10b-9d73211cdf30:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d323fe12-2cfb-4cc7-a99b-7664220f4256:/dev/nvme0n1p3] Nov 04 02:31:49.152055 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.151955 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 02:31:49.157805 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.157668 2575 manager.go:217] Machine: {Timestamp:2026-11-04 02:31:49.156465298 +0000 UTC m=+0.334242237 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101102 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269919f03f05cdeaf4c668bb5a23fb SystemUUID:ec269919-f03f-05cd-eaf4-c668bb5a23fb BootID:87d03c5d-3e5a-449e-86c1-f3f879f0fb29 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:49:00:d0:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:49:00:d0:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:95:c9:b0:03:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 02:31:49.157805 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.157792 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 02:31:49.158017 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.157978 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 02:31:49.160620 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.160594 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 02:31:49.160779 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.160621 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-209.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 02:31:49.161366 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.161355 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 02:31:49.161404 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.161367 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 02:31:49.161404 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.161381 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:31:49.162259 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.162246 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:31:49.163012 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.163000 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 02:31:49.166938 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.166922 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 02:31:49.167116 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167107 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 02:31:49.167869 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167858 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 04 02:31:49.167912 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167877 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 02:31:49.167912 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167889 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 02:31:49.167990 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167924 2575 kubelet.go:405] "Adding apiserver pod source" Nov 04 02:31:49.167990 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.167933 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 02:31:49.168438 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.168421 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:31:49.170685 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.170668 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 02:31:49.177034 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177014 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:31:49.177590 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177575 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 02:31:49.177661 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177606 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 02:31:49.177768 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177757 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 02:31:49.177818 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177772 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 02:31:49.177818 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177779 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 02:31:49.177818 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177784 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 02:31:49.177818 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177790 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 02:31:49.177818 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.177796 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 02:31:49.178561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178551 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 02:31:49.178561 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178561 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 02:31:49.178625 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178568 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 02:31:49.178625 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178574 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 02:31:49.178625 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178582 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 02:31:49.178625 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.178592 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 02:31:49.179299 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.179289 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 02:31:49.179299 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.179299 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 02:31:49.183216 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.183193 2575 server.go:1263] "Started kubelet" Nov 04 02:31:49.183344 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.183313 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 02:31:49.183415 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.183311 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 02:31:49.183476 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.183413 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 02:31:49.184198 ip-10-0-133-209 systemd[1]: Started Kubernetes Kubelet. Nov 04 02:31:49.185157 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.185145 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 04 02:31:49.187759 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.187738 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 02:31:49.189845 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.189824 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 02:31:49.191843 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.191829 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 02:31:49.191909 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.191900 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 02:31:49.192173 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.192126 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:31:49.192515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.192454 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 02:31:49.192515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.192504 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 02:31:49.192635 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.192521 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 02:31:49.192786 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.192751 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 04 02:31:49.192838 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.192780 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 04 02:31:49.193142 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193124 2575 factory.go:55] Registering systemd factory Nov 04 02:31:49.193233 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193145 2575 factory.go:223] Registration of the systemd container factory successfully Nov 04 02:31:49.193396 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193379 2575 factory.go:153] Registering CRI-O factory Nov 04 02:31:49.193467 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193402 2575 factory.go:223] Registration of the crio container factory successfully Nov 04 02:31:49.193467 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193459 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 02:31:49.193571 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193485 2575 factory.go:103] Registering Raw factory Nov 04 02:31:49.193571 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.193500 2575 manager.go:1199] Started watching for new ooms in manager Nov 04 02:31:49.194651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.194635 2575 manager.go:319] Starting recovery of all containers Nov 04 02:31:49.208688 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.208670 2575 manager.go:324] Recovery completed Nov 04 02:31:49.212798 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.212784 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.214445 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.214422 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.214515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.214457 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.214515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.214469 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.215013 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.214997 2575 cpu_manager.go:226] "Starting" policy="none" Nov 04 02:31:49.215013 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215011 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 02:31:49.215138 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215034 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 02:31:49.215138 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215045 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.215730 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215715 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.215809 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215746 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.215809 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.215761 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.217261 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.217247 2575 policy_none.go:50] "Start" Nov 04 02:31:49.217331 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.217263 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 02:31:49.217331 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.217293 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 02:31:49.219342 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.219331 2575 policy_none.go:44] "Start" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.262634 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.266940 2575 manager.go:341] "Starting Device Plugin manager" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.266991 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267006 2575 server.go:87] "Starting device plugin registration server" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267243 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267255 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267362 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267442 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.267455 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.268054 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.268117 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.274759 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.276216 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.276231 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.276259 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 02:31:49.281684 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.276311 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 02:31:49.328587 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.327736 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-209.us-west-1.compute.internal.18e4b6c51557662a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-209.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-209.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 02:31:49.183161898 +0000 UTC m=+0.360938838,LastTimestamp:2026-11-04 02:31:49.183161898 +0000 UTC m=+0.360938838,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-209.us-west-1.compute.internal,}" Nov 04 02:31:49.368298 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.368240 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.369382 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.369363 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.369505 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.369408 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.369505 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.369422 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.369505 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.369459 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.375936 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.375913 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-209.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 02:31:49.376038 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.375977 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-209.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 02:31:49.377142 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.377126 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal"] Nov 04 02:31:49.377207 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.377197 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.378069 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.378054 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.378132 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.378088 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.378132 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.378105 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.380293 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.380267 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.380451 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.380422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.380486 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.380468 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.381232 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381208 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.381341 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381244 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.381341 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381254 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.381424 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381406 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.381466 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381438 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.381466 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.381452 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.384099 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.384082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.384168 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.384129 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:49.384833 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.384815 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:49.384903 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.384864 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:49.384903 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.384880 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:49.386197 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.386111 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-209.us-west-1.compute.internal.18e4b6c51734ac99 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-209.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-209.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-209.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 02:31:49.214440601 +0000 UTC m=+0.392217541,LastTimestamp:2026-11-04 02:31:49.214440601 +0000 UTC m=+0.392217541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-209.us-west-1.compute.internal,}" Nov 04 02:31:49.386711 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.386690 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 04 02:31:49.393099 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.393075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.393160 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.393109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.393193 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.393158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.404744 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.404721 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 02:31:49.404808 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.404781 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnjvb" Nov 04 02:31:49.413788 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.413769 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.416555 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.416538 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.488226 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.488196 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:49.488891 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.488873 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:49.493391 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.493475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.493475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493434 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.493475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493449 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.493475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.493607 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.493485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.588318 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.588213 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.588318 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:49.588257 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-209.us-west-1.compute.internal\": node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:31:49.588677 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.588659 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:49.592589 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.592573 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.656479 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.656450 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:31:49.657347 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.657329 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.715268 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.715240 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:31:49.717317 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.717298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:49.719465 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:49.719429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:31:50.122667 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.122642 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 02:31:50.123195 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.122815 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 02:31:50.123195 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.122846 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 02:31:50.123195 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.122795 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7878fea1d4a142dfa615cdc355a2760-4128f5fe3c6c09e5.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-209.us-west-1.compute.internal.18e4b6c517352b0a\": read tcp 10.0.133.209:37480->18.227.234.46:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-209.us-west-1.compute.internal.18e4b6c517352b0a default 4966 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-209.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-209.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-209.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 02:31:49 +0000 UTC,LastTimestamp:2026-11-04 02:31:49.369428556 +0000 UTC m=+0.547205498,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-209.us-west-1.compute.internal,}" Nov 04 02:31:50.168767 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.168739 2575 apiserver.go:52] "Watching apiserver" Nov 04 02:31:50.192639 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.192602 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 02:31:50.409608 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.409499 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 02:26:51 +0000 UTC" deadline="2028-06-18 08:35:13.378322213 +0000 UTC" Nov 04 02:31:50.409608 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.409560 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14214h3m22.968766766s" Nov 04 02:31:50.429857 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.429834 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 02:31:50.430176 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.430154 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-k6jrq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g","openshift-multus/multus-additional-cni-plugins-rsmrc","openshift-multus/network-metrics-daemon-mbppk","kube-system/konnectivity-agent-qm2lj","kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal","openshift-dns/node-resolver-8m8pf","openshift-image-registry/node-ca-txm72","openshift-ovn-kubernetes/ovnkube-node-82r7r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-4xfw5","openshift-network-operator/iptables-alerter-mcv6p","kube-system/global-pull-secret-syncer-65l2h","openshift-cluster-node-tuning-operator/tuned-wrn76"] Nov 04 02:31:50.432626 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.432611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.434729 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.434710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.436764 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.436749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.438733 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.438720 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.438791 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.438775 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:31:50.440701 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.440685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.442754 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.442741 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.444768 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.444754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.447070 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.447054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.449147 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.449131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:50.449218 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.449194 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:31:50.453029 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.453010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.455288 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.455264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.455351 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.455337 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:31:50.457351 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.457334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.485478 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.485460 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ffw8m\"" Nov 04 02:31:50.486211 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486190 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 02:31:50.486613 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486564 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 02:31:50.486713 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486674 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.486713 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486690 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.486713 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486711 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.486846 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486721 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 02:31:50.486846 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486745 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm8qv\"" Nov 04 02:31:50.486846 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.486776 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.488009 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.487996 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 02:31:50.488385 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.488373 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgzgm\"" Nov 04 02:31:50.488427 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.488383 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 02:31:50.493370 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.493355 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 02:31:50.497493 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.497665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.497929 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.497980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:50.498166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9vn\" (UniqueName: \"kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566wr\" (UniqueName: \"kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.498601 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrspf\" (UniqueName: \"kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.498991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9plm\" (UniqueName: \"kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499191 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwdf7\" (UniqueName: \"kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tmv\" (UniqueName: \"kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.499862 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwhl\" (UniqueName: \"kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxsj8\" (UniqueName: \"kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pvs\" (UniqueName: \"kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.500305 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.499843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.589737 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589704 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 02:31:50.589889 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589747 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 02:31:50.589889 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589852 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.590007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589939 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.590007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589959 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4v2qx\"" Nov 04 02:31:50.590007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589976 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6l7l7\"" Nov 04 02:31:50.590007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.589985 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.590181 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590012 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.590181 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590085 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.590181 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590155 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 02:31:50.590352 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590231 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.590352 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590271 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.590352 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590310 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vd7gf\"" Nov 04 02:31:50.590352 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590344 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.590558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590379 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 02:31:50.590558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590427 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 02:31:50.590558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590456 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 02:31:50.590558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590520 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpvh8" Nov 04 02:31:50.590760 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590580 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 02:31:50.590760 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590595 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 02:31:50.590760 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590663 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjd4x\"" Nov 04 02:31:50.590903 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590888 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 02:31:50.591009 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.590991 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmwv6\"" Nov 04 02:31:50.591139 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.591122 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 02:31:50.591208 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.591192 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 02:31:50.591384 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.591350 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lg9c8\"" Nov 04 02:31:50.600720 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.600814 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9vn\" (UniqueName: \"kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.600814 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566wr\" (UniqueName: \"kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.600814 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.600814 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.600814 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.600991 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.601022 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.600996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601022 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.601088 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601088 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601088 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601088 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601088 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrspf\" (UniqueName: \"kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601246 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9plm\" (UniqueName: \"kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601454 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601495 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.601174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.601722 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.601621 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:50.602488 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.602191 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:51.102170601 +0000 UTC m=+2.279947545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:50.602488 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.602231 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.602488 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.602266 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.602488 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.602310 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.602672 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.602539 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.603067 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.602981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.604686 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.604667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.604896 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.604873 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.605017 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.604993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.608844 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.608917 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.608917 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.608917 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.608980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwdf7\" (UniqueName: \"kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tmv\" (UniqueName: \"kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609142 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609203 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609244 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwhl\" (UniqueName: \"kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxsj8\" (UniqueName: \"kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609418 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pvs\" (UniqueName: \"kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609524 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609531 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609584 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609594 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609690 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.609964 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.609985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.610558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610263 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610354 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.611080 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610395 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610428 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.610409 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.610475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:31:51.110462659 +0000 UTC m=+2.288239607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610532 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610584 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610708 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.611651 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610770 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610804 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.610840 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.611330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.611336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.611576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.611604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.612145 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.611638 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.612397 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.612144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:50.613187 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.613171 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.662345 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.662260 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9vn\" (UniqueName: \"kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:50.662345 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.662267 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrspf\" (UniqueName: \"kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:50.662690 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.662669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566wr\" (UniqueName: \"kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:50.663469 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.663435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9plm\" (UniqueName: \"kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:50.669504 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.669480 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:50.669631 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.669511 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:50.669631 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.669526 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:50.669631 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:50.669621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:51.169587475 +0000 UTC m=+2.347364639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:50.672363 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.672334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwdf7\" (UniqueName: \"kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:50.672499 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.672362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tmv\" (UniqueName: \"kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:31:50.672559 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.672489 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxsj8\" (UniqueName: \"kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:50.672909 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.672887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwhl\" (UniqueName: \"kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:50.674203 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.674173 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pvs\" (UniqueName: \"kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:50.732036 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:50.732013 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:52.487232 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.487207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6jrq" Nov 04 02:31:52.501574 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.501527 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0ca244_6231_4058_9051_8aa91ae0ce73.slice/crio-8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a WatchSource:0}: Error finding container 8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a: Status 404 returned error can't find the container with id 8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a Nov 04 02:31:52.502108 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.502086 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0663bb7aee2243f0f782ca4003b09473.slice/crio-41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f WatchSource:0}: Error finding container 41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f: Status 404 returned error can't find the container with id 41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f Nov 04 02:31:52.502715 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.502698 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41843a288696f78001f75198b2c4caf0.slice/crio-4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41 WatchSource:0}: Error finding container 4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41: Status 404 returned error can't find the container with id 4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41 Nov 04 02:31:52.506890 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.506858 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:31:52.508620 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.508598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:31:52.517636 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.517614 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a5e4c2_b127_4bfe_9a5b_943e48410287.slice/crio-3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9 WatchSource:0}: Error finding container 3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9: Status 404 returned error can't find the container with id 3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9 Nov 04 02:31:52.531819 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.531789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:31:52.536002 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.535802 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:31:52.543655 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.543627 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d950c5_819d_4cc5_aa08_7f80bf099eb1.slice/crio-f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7 WatchSource:0}: Error finding container f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7: Status 404 returned error can't find the container with id f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7 Nov 04 02:31:52.544054 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.544033 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2376a8d_13b4_490b_9f66_b55cf5f3a51f.slice/crio-a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf WatchSource:0}: Error finding container a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf: Status 404 returned error can't find the container with id a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf Nov 04 02:31:52.558044 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.558018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:31:52.567664 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.567637 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af83a8_adea_4abe_962e_c95203bdcde2.slice/crio-a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59 WatchSource:0}: Error finding container a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59: Status 404 returned error can't find the container with id a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59 Nov 04 02:31:52.577040 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.577011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-txm72" Nov 04 02:31:52.586571 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.586538 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864630d2_d25b_4d12_aa4a_c9a60f5a13f4.slice/crio-b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d WatchSource:0}: Error finding container b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d: Status 404 returned error can't find the container with id b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d Nov 04 02:31:52.607315 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.607285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:31:52.616212 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.616178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:31:52.617058 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.617033 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399e71a6_e15f_4aea_b7fc_25f45d6c182e.slice/crio-4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138 WatchSource:0}: Error finding container 4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138: Status 404 returned error can't find the container with id 4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138 Nov 04 02:31:52.620712 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.620691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:31:52.624827 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.624801 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584afc7e_72fd_4893_a77c_519e4b0b44ad.slice/crio-208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa WatchSource:0}: Error finding container 208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa: Status 404 returned error can't find the container with id 208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa Nov 04 02:31:52.629397 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:31:52.629373 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abdefaf_ed85_4c17_bfe7_11b0aec18b44.slice/crio-0e4338e41a9c6cbcc74c9fd341ab38267c6ceeafb5050195f9c8687e9e995fa2 WatchSource:0}: Error finding container 0e4338e41a9c6cbcc74c9fd341ab38267c6ceeafb5050195f9c8687e9e995fa2: Status 404 returned error can't find the container with id 0e4338e41a9c6cbcc74c9fd341ab38267c6ceeafb5050195f9c8687e9e995fa2 Nov 04 02:31:52.631306 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.631287 2575 kuberuntime_manager.go:1707] "Unhandled Error" err="container tuned start failed in pod tuned-wrn76_openshift-cluster-node-tuning-operator(0abdefaf-ed85-4c17-bfe7-11b0aec18b44): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 04 02:31:52.631349 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.631319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" podUID="0abdefaf-ed85-4c17-bfe7-11b0aec18b44" Nov 04 02:31:52.834712 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.834679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:52.834899 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.834727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:52.834899 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.834824 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:52.834899 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.834828 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:52.834899 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.834886 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:53.834872768 +0000 UTC m=+3.292737595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:52.835038 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.834904 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:31:53.834898196 +0000 UTC m=+3.292763023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:52.935269 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.935230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:52.935996 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.935380 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:52.935996 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.935395 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:52.935996 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.935404 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:52.935996 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:52.935450 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:53.935437846 +0000 UTC m=+3.393302675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:52.964987 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:52.964962 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:53.001147 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.001091 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mcv6p" event={"ID":"584afc7e-72fd-4893-a77c-519e4b0b44ad","Type":"ContainerStarted","Data":"208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa"} Nov 04 02:31:53.002259 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.002237 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8m8pf" event={"ID":"78af83a8-adea-4abe-962e-c95203bdcde2","Type":"ContainerStarted","Data":"a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59"} Nov 04 02:31:53.003281 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.003253 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerStarted","Data":"f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7"} Nov 04 02:31:53.004295 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.004273 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" event={"ID":"82a5e4c2-b127-4bfe-9a5b-943e48410287","Type":"ContainerStarted","Data":"3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9"} Nov 04 02:31:53.005297 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.005277 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" event={"ID":"0abdefaf-ed85-4c17-bfe7-11b0aec18b44","Type":"ContainerStarted","Data":"0e4338e41a9c6cbcc74c9fd341ab38267c6ceeafb5050195f9c8687e9e995fa2"} Nov 04 02:31:53.006528 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.006509 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138"} Nov 04 02:31:53.007254 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.007224 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" podUID="0abdefaf-ed85-4c17-bfe7-11b0aec18b44" Nov 04 02:31:53.007621 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.007597 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-txm72" event={"ID":"864630d2-d25b-4d12-aa4a-c9a60f5a13f4","Type":"ContainerStarted","Data":"b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d"} Nov 04 02:31:53.008612 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.008583 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qm2lj" event={"ID":"b2376a8d-13b4-490b-9f66-b55cf5f3a51f","Type":"ContainerStarted","Data":"a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf"} Nov 04 02:31:53.009543 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.009518 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6jrq" event={"ID":"cb0ca244-6231-4058-9051-8aa91ae0ce73","Type":"ContainerStarted","Data":"8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a"} Nov 04 02:31:53.010648 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.010628 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerStarted","Data":"41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f"} Nov 04 02:31:53.011583 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.011563 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"41843a288696f78001f75198b2c4caf0","Type":"ContainerStarted","Data":"4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41"} Nov 04 02:31:53.254956 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.254918 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:53.314996 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.314956 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:52 +0000 UTC" deadline="2028-06-19 22:43:49.799452319 +0000 UTC" Nov 04 02:31:53.314996 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.314986 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14252h11m56.484469556s" Nov 04 02:31:53.843978 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.843935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:53.844166 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.844002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:53.844166 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.844162 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:53.844309 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.844243 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.844225174 +0000 UTC m=+5.302090015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:53.844370 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.844315 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:53.844370 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.844347 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.8443368 +0000 UTC m=+5.302201640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:53.944612 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.944572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:53.945096 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.944739 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:53.945096 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.944762 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:53.945096 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.944775 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:53.945096 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.944831 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.944814967 +0000 UTC m=+5.402679815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:53.996846 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.996814 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:53.997041 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.996980 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:31:53.997130 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.997120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:53.997258 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.997227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:31:53.997367 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:53.997322 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:53.997427 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:53.997408 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:31:54.315672 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:54.315635 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:52 +0000 UTC" deadline="2028-07-18 00:07:32.374421712 +0000 UTC" Nov 04 02:31:54.315672 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:54.315670 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14925h35m38.058756501s" Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.859371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.859468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.859598 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.859660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:59.859642818 +0000 UTC m=+9.317507648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.859807 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.860105 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.859842 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:31:59.859831033 +0000 UTC m=+9.317695874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.961746 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.961682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:55.963074 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.962834 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:55.963074 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.962861 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:55.963074 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.962875 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:55.963074 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.963033 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:59.962999169 +0000 UTC m=+9.420864010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:55.996979 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.996755 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:55.996979 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.996891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:55.996979 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.996886 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:31:55.997402 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.997361 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:31:55.997402 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:55.997401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:55.997568 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:55.997492 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:31:57.034491 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.034455 2575 generic.go:356] "Generic (PLEG): container finished" podID="0663bb7aee2243f0f782ca4003b09473" containerID="da789e0bc777e5dae368397004b4c5074f8318cacfa5909e51de27cd47f304e2" exitCode=0 Nov 04 02:31:57.034961 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.034561 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerDied","Data":"da789e0bc777e5dae368397004b4c5074f8318cacfa5909e51de27cd47f304e2"} Nov 04 02:31:57.046033 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.045997 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"41843a288696f78001f75198b2c4caf0","Type":"ContainerStarted","Data":"36077d532873c0f9344963fd8a1ab8f8bc6e54c61d5c3cc21248058f3f31abbf"} Nov 04 02:31:57.052888 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.052854 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="f8a443a540fdb2d7db5713b36538e66bb0e5d307196e85fdd75caa4a74e81008" exitCode=0 Nov 04 02:31:57.053016 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.052912 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"f8a443a540fdb2d7db5713b36538e66bb0e5d307196e85fdd75caa4a74e81008"} Nov 04 02:31:57.264350 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.263805 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" podStartSLOduration=6.263783932 podStartE2EDuration="6.263783932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:31:57.262916749 +0000 UTC m=+6.720781595" watchObservedRunningTime="2026-11-04 02:31:57.263783932 +0000 UTC m=+6.721648782" Nov 04 02:31:57.996810 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.996757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:57.996988 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:57.996916 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:31:57.997360 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.997330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:57.997463 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:57.997445 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:31:57.997517 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:57.997498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:57.997593 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:57.997570 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:31:58.060949 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:58.060908 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerStarted","Data":"5a324a9ec436b3dafc983669fb6e4e233a9fbe2a8b687fce1093359ff621c4e6"} Nov 04 02:31:59.064690 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.064647 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-txm72" event={"ID":"864630d2-d25b-4d12-aa4a-c9a60f5a13f4","Type":"ContainerStarted","Data":"31106e5ab9378eddd18b59e654e59b78419e8be60b7ed72444812b92ca4324f1"} Nov 04 02:31:59.067354 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.067321 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qm2lj" event={"ID":"b2376a8d-13b4-490b-9f66-b55cf5f3a51f","Type":"ContainerStarted","Data":"5a6b183fe9edef41580d53dd71e7029e3b7111ffe6a9601e2e4280ce4a9244e5"} Nov 04 02:31:59.070535 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.070507 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" event={"ID":"82a5e4c2-b127-4bfe-9a5b-943e48410287","Type":"ContainerStarted","Data":"bea737059d1977a450054afe1bb48eb6a13ae3040b1fffccc864ec49c49d627b"} Nov 04 02:31:59.179519 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.179457 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-txm72" podStartSLOduration=2.6274384619999998 podStartE2EDuration="8.179438761s" totalImagesPullingTime="5.552000299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:31:59.179278089 +0000 UTC m=+8.637142950" watchObservedRunningTime="2026-11-04 02:31:59.179438761 +0000 UTC m=+8.637303612" Nov 04 02:31:59.179779 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.179749 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" podStartSLOduration=8.179742449999999 podStartE2EDuration="8.17974245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:31:58.18292312 +0000 UTC m=+7.640787970" watchObservedRunningTime="2026-11-04 02:31:59.17974245 +0000 UTC m=+8.637607302" Nov 04 02:31:59.902788 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.902705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:59.902950 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.902803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:59.902950 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.902930 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:59.903057 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.902994 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:07.902974995 +0000 UTC m=+17.360839840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:59.903124 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.903076 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:59.903124 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.903115 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:07.90310114 +0000 UTC m=+17.360965981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.997007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.997157 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.997262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.997350 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:31:59.997405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:31:59.997514 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:31:59.997470 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:00.003408 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:00.003370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:00.003626 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:00.003604 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:32:00.003712 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:00.003629 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:32:00.003712 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:00.003642 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:00.003712 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:00.003705 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:08.003685571 +0000 UTC m=+17.461550421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:00.958989 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:00.958899 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:00.959961 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:00.959943 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:01.073890 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.073840 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qm2lj" podStartSLOduration=4.478319374 podStartE2EDuration="10.073825928s" totalImagesPullingTime="5.595506554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:31:59.294504675 +0000 UTC m=+8.752369524" watchObservedRunningTime="2026-11-04 02:32:01.073825928 +0000 UTC m=+10.531690778" Nov 04 02:32:01.075713 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.075663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6jrq" event={"ID":"cb0ca244-6231-4058-9051-8aa91ae0ce73","Type":"ContainerStarted","Data":"2d8a52553f53926a98dea06ebba3cbd08149fb8ecb2fd3d6761c2d3f39298e99"} Nov 04 02:32:01.077074 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.077044 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8m8pf" event={"ID":"78af83a8-adea-4abe-962e-c95203bdcde2","Type":"ContainerStarted","Data":"c2ee3cdb1dbfb347134a8dbdf0e9e5a3a579656d0cd4033fcf02790211fa814a"} Nov 04 02:32:01.078693 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.078671 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="f0674b0ba8a9fb4c6c32dffda75bb6967f1cc894d88e414d33901f7f8f5452e5" exitCode=0 Nov 04 02:32:01.078792 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.078744 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"f0674b0ba8a9fb4c6c32dffda75bb6967f1cc894d88e414d33901f7f8f5452e5"} Nov 04 02:32:01.244835 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.244729 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k6jrq" podStartSLOduration=2.337782089 podStartE2EDuration="10.244714063s" totalImagesPullingTime="7.906931974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:01.244287685 +0000 UTC m=+10.702152537" watchObservedRunningTime="2026-11-04 02:32:01.244714063 +0000 UTC m=+10.702578920" Nov 04 02:32:01.480461 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.480400 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8m8pf" podStartSLOduration=3.092053718 podStartE2EDuration="10.480383415s" totalImagesPullingTime="7.388329697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:01.479999913 +0000 UTC m=+10.937864762" watchObservedRunningTime="2026-11-04 02:32:01.480383415 +0000 UTC m=+10.938248265" Nov 04 02:32:01.997224 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.997020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:01.997682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:01.997334 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:01.997682 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.997037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:01.997682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:01.997428 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:01.997682 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:01.997018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:01.997682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:01.997506 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:02.050357 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.050321 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 02:32:02.083071 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.083035 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mcv6p" event={"ID":"584afc7e-72fd-4893-a77c-519e4b0b44ad","Type":"ContainerStarted","Data":"ab32fb786c98b3af59cf0ed21da54f96890fc43ba52d32eac0fa7be4ec40036b"} Nov 04 02:32:02.085168 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.085130 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" event={"ID":"82a5e4c2-b127-4bfe-9a5b-943e48410287","Type":"ContainerStarted","Data":"91255deb8d1756c5dc25c32cce7b353497e9ec9fd564d6998f316ef99cd09551"} Nov 04 02:32:02.085320 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.085175 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 02:32:02.197162 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.197114 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mcv6p" podStartSLOduration=3.418792467 podStartE2EDuration="11.197099752s" totalImagesPullingTime="7.778307285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:02.196963003 +0000 UTC m=+11.654827852" watchObservedRunningTime="2026-11-04 02:32:02.197099752 +0000 UTC m=+11.654964600" Nov 04 02:32:02.994448 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.994330 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T02:32:02.050343425Z","UUID":"1f549979-2e77-4888-9f15-ce23cf4f8d69","Handler":null,"Name":"","Endpoint":""} Nov 04 02:32:02.996105 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.996083 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 02:32:02.996268 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:02.996115 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 02:32:03.090541 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:03.090504 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="1f1cc8091632e0ac27fe25325632b6e980091b712f3a67cdbf2dca7dae27b63c" exitCode=0 Nov 04 02:32:03.091025 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:03.090595 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"1f1cc8091632e0ac27fe25325632b6e980091b712f3a67cdbf2dca7dae27b63c"} Nov 04 02:32:03.997346 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:03.997313 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:03.997520 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:03.997381 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:03.997520 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:03.997415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:03.997628 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:03.997547 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:03.997628 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:03.997595 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:03.997721 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:03.997701 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:04.094940 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:04.094896 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" event={"ID":"82a5e4c2-b127-4bfe-9a5b-943e48410287","Type":"ContainerStarted","Data":"2ca2dc9c4806437a12f6e7beb812d7198a401f918544bacd83321759cf26e105"} Nov 04 02:32:04.212976 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:04.212921 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" podStartSLOduration=2.120361169 podStartE2EDuration="13.212906622s" totalImagesPullingTime="11.092545453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:04.212825136 +0000 UTC m=+13.670689991" watchObservedRunningTime="2026-11-04 02:32:04.212906622 +0000 UTC m=+13.670771472" Nov 04 02:32:05.099628 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:05.099589 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="6a09ac27ff7569ae5beb339d46ad0b6a5b6ecd143d007ac3d06e29a71954603c" exitCode=0 Nov 04 02:32:05.100247 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:05.099673 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"6a09ac27ff7569ae5beb339d46ad0b6a5b6ecd143d007ac3d06e29a71954603c"} Nov 04 02:32:05.997079 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:05.997032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:05.997276 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:05.997168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:05.997276 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:05.997255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:05.997396 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:05.997317 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:05.997396 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:05.997372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:05.997497 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:05.997432 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:07.965923 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:07.965886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:07.966646 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:07.965949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:07.966646 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.966048 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:07.966646 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.966083 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:07.966646 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.966123 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.966109171 +0000 UTC m=+33.423973998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:07.966646 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.966146 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret podName:95e3e816-aa1d-4022-b606-0b4ecc1ee79e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.966129785 +0000 UTC m=+33.423994638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret") pod "global-pull-secret-syncer-65l2h" (UID: "95e3e816-aa1d-4022-b606-0b4ecc1ee79e") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:07.997157 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:07.997125 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:07.997368 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:07.997265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:07.997368 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.997280 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:07.997635 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.997606 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:07.997767 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:07.997675 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:07.997823 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:07.997762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:08.067239 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:08.066687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:08.067239 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:08.066853 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:32:08.067239 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:08.066880 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:32:08.067239 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:08.066893 2575 projected.go:196] Error preparing data for projected volume kube-api-access-8g9f9 for pod openshift-network-diagnostics/network-check-target-4xfw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:08.067239 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:08.066955 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9 podName:59ccac4d-0b67-439b-a462-e11f04cdb14c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.066935845 +0000 UTC m=+33.524800678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8g9f9" (UniqueName: "kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9") pod "network-check-target-4xfw5" (UID: "59ccac4d-0b67-439b-a462-e11f04cdb14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:09.112545 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:09.112341 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="db788d61a7d23e02451ac5eadc0862e6ca70ff6165e8e028e466dbdab3127cc5" exitCode=0 Nov 04 02:32:09.113014 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:09.112423 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"db788d61a7d23e02451ac5eadc0862e6ca70ff6165e8e028e466dbdab3127cc5"} Nov 04 02:32:09.996805 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:09.996774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:09.996805 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:09.996788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:09.997020 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:09.996819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:09.997020 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:09.996910 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:09.997020 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:09.996996 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:09.997159 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:09.997102 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:10.119709 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.119679 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:10.120341 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120312 2575 generic.go:356] "Generic (PLEG): container finished" podID="399e71a6-e15f-4aea-b7fc-25f45d6c182e" containerID="bff232e2875d6d3346bea1674f1376e6baff81a3c20a757332e8b96c014a0b51" exitCode=1 Nov 04 02:32:10.120453 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120421 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"e7acc47ac537d25269059cf2b3c71a9b2ba8dc87ff18e26e7263ebf986bd0fda"} Nov 04 02:32:10.120526 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120504 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"b4e8cb87d7dadb8f54d8789cdcaabd9a97cac2dbaa76c1fef042e2ec0cd75396"} Nov 04 02:32:10.120582 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120553 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"2382f330fb2cb10b3efe90d6d4caaa0386f740e89706eb37e93c6033599aa9d1"} Nov 04 02:32:10.120638 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120602 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"c8672c669f4889b83d41cc22e34f559a3c268885a52734da27d2918cefbcd2e7"} Nov 04 02:32:10.120686 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120652 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerDied","Data":"bff232e2875d6d3346bea1674f1376e6baff81a3c20a757332e8b96c014a0b51"} Nov 04 02:32:10.120733 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.120703 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"47cfe8823c1fa36586e97797338d35e4d141e31daed788baca97c8d4ee2d5706"} Nov 04 02:32:10.124678 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.124649 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="e0889175e57a076873f6d7303ef3fc57c310579a0dad98390414ec2d1c5e1236" exitCode=0 Nov 04 02:32:10.124808 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:10.124691 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerDied","Data":"e0889175e57a076873f6d7303ef3fc57c310579a0dad98390414ec2d1c5e1236"} Nov 04 02:32:11.131270 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:11.131052 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" event={"ID":"a5d950c5-819d-4cc5-aa08-7f80bf099eb1","Type":"ContainerStarted","Data":"24b5ca52cacd9d348ae5c69205b270ce374c44cbd832b2cdcf22dd40333010db"} Nov 04 02:32:11.252739 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:11.252681 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rsmrc" podStartSLOduration=7.859111056 podStartE2EDuration="20.252660619s" totalImagesPullingTime="12.393549563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:11.252629508 +0000 UTC m=+20.710494358" watchObservedRunningTime="2026-11-04 02:32:11.252660619 +0000 UTC m=+20.710525465" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:11.996858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:11.996870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:11.996977 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:11.996997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:11.997099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:11.997328 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:11.997207 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:13.137592 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.137544 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" event={"ID":"0abdefaf-ed85-4c17-bfe7-11b0aec18b44","Type":"ContainerStarted","Data":"b166bbb320af9f4a6b031d26a9ca17cd6bf77d93f8e90e0f8b52a48d0338c65d"} Nov 04 02:32:13.140673 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.140653 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:13.140993 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.140974 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"5e36856d417c2ac5a2f91c24116e0c99286d94761c90771488d6be5c6bda53a7"} Nov 04 02:32:13.272025 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.271922 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wrn76" podStartSLOduration=1.9079150839999999 podStartE2EDuration="22.271908828s" totalImagesPullingTime="20.363993744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-04 02:32:13.271722442 +0000 UTC m=+22.729587302" watchObservedRunningTime="2026-11-04 02:32:13.271908828 +0000 UTC m=+22.729773677" Nov 04 02:32:13.997096 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.997066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:13.997280 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.997104 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:13.997280 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:13.997159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:13.997280 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:13.997178 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:13.997280 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:13.997244 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:13.997402 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:13.997307 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:15.149805 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.149774 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:15.150685 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.150210 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"868ac66111e1a27e27fa51ec251c73aa3b133bffab3a4de45dd3929d7e408e5e"} Nov 04 02:32:15.150685 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.150502 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:15.150685 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.150563 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:15.150685 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.150613 2575 scope.go:122] "RemoveContainer" containerID="bff232e2875d6d3346bea1674f1376e6baff81a3c20a757332e8b96c014a0b51" Nov 04 02:32:15.168543 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.168473 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:15.996828 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.996624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:15.997003 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.996648 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:15.997003 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:15.996917 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:15.997003 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:15.996690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:15.997003 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:15.996977 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:15.997132 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:15.997046 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:16.155855 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.155826 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:16.156291 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.156172 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" event={"ID":"399e71a6-e15f-4aea-b7fc-25f45d6c182e","Type":"ContainerStarted","Data":"e98960ce9b8fd9728943a28073aaa9fe91d41c713d49a0ca03741c16e500fac7"} Nov 04 02:32:16.156396 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.156377 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:16.172442 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.172413 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:16.288136 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.288040 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" podStartSLOduration=8.210713964 podStartE2EDuration="25.288027136s" totalImagesPullingTime="17.077313172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:16.287470466 +0000 UTC m=+25.745335317" watchObservedRunningTime="2026-11-04 02:32:16.288027136 +0000 UTC m=+25.745891986" Nov 04 02:32:16.983738 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.983708 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xfw5"] Nov 04 02:32:16.983908 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.983805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:16.983908 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:16.983898 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:16.988534 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.988508 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65l2h"] Nov 04 02:32:16.988660 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.988611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:16.988732 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:16.988712 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:16.991680 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.991658 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbppk"] Nov 04 02:32:16.991786 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:16.991744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:16.991853 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:16.991838 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:18.348098 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.348066 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:18.348597 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.348224 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 02:32:18.348864 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.348847 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:18.997106 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.997062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:18.997106 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.997086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:18.997361 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:18.997280 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:18.997361 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:18.997320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:18.997457 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:18.997428 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:18.997513 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:18.997489 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:20.997412 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:20.997382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:20.997965 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:20.997490 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xfw5" podUID="59ccac4d-0b67-439b-a462-e11f04cdb14c" Nov 04 02:32:20.997965 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:20.997500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:20.997965 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:20.997595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:20.997965 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:20.997596 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbppk" podUID="220ee58a-5ca9-4f09-a317-e5b66524efb0" Nov 04 02:32:20.997965 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:20.997657 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65l2h" podUID="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" Nov 04 02:32:22.335912 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.335887 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeReady" Nov 04 02:32:22.336351 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.336015 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 02:32:22.440155 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.440118 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8xkb7"] Nov 04 02:32:22.480012 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.479973 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cz2p"] Nov 04 02:32:22.480177 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.480094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:22.493438 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.493419 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzwxn"] Nov 04 02:32:22.493553 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.493536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:22.506337 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.506314 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6"] Nov 04 02:32:22.506457 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.506417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:22.530787 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.530751 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l2nk6"] Nov 04 02:32:22.530954 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.530886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:22.547003 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.546963 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:32:22.547003 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.546971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.551454 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.551421 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.551615 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.551502 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.564772 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.564723 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2"] Nov 04 02:32:22.565174 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.564836 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.568570 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.568539 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9"] Nov 04 02:32:22.568705 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.568656 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.581987 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.581924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfchl\" (UniqueName: \"kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl\") pod \"kas-connection-checker-685895c7c9-8cz2p\" (UID: \"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed\") " pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:22.582126 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.582003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7qp\" (UniqueName: \"kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp\") pod \"kas-connection-checker-685895c7c9-8xkb7\" (UID: \"64d03902-4d33-4bec-9725-53ec99ca7dbd\") " pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:22.587209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.587152 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76598c4567-mhntk"] Nov 04 02:32:22.587356 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.587310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.599301 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.599278 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 02:32:22.599557 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.599539 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mk8j7\"" Nov 04 02:32:22.601040 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.601025 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2"] Nov 04 02:32:22.601160 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.601145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.625329 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.625290 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nxqpl"] Nov 04 02:32:22.625505 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.625421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.639405 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.639371 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:32:22.639534 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.639483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.653572 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.653548 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zp9vk"] Nov 04 02:32:22.653706 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.653669 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.656693 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656661 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.656693 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656661 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.656927 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656911 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 02:32:22.657055 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656966 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.657055 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656987 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.657055 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.656996 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.657055 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.657034 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 02:32:22.657276 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.657069 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 02:32:22.657276 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.657114 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.657276 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.657112 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.657426 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.657382 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.676389 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.676363 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 02:32:22.677722 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.677704 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn"] Nov 04 02:32:22.677847 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.677830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.682637 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.682745 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwq9\" (UniqueName: \"kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.682745 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.682866 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.682866 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.682866 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.682866 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.682991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.683073 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptjw\" (UniqueName: \"kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfchl\" (UniqueName: \"kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl\") pod \"kas-connection-checker-685895c7c9-8cz2p\" (UID: \"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed\") " pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7qp\" (UniqueName: \"kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp\") pod \"kas-connection-checker-685895c7c9-8xkb7\" (UID: \"64d03902-4d33-4bec-9725-53ec99ca7dbd\") " pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwfc\" (UniqueName: \"kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc\") pod \"volume-data-source-validator-58c9f5f46d-2chk6\" (UID: \"e5ee7cb7-c423-4701-b7ab-3ee603748499\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:22.683529 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.683414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbjk\" (UniqueName: \"kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk\") pod \"kas-connection-checker-685895c7c9-bzwxn\" (UID: \"6e9a6a6d-33fc-45ea-9b03-af9724367475\") " pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:22.691906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.691882 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc"] Nov 04 02:32:22.692017 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.691991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.706058 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.706037 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-89rsm"] Nov 04 02:32:22.706234 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.706217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:22.707002 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.706977 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 02:32:22.707099 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707042 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.707158 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707123 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.707158 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707137 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.707306 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707293 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 02:32:22.707553 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707537 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.707648 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.707603 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 02:32:22.708398 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.708380 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 02:32:22.709914 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709879 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 02:32:22.709914 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709906 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709933 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709936 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709958 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709965 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.709972 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 02:32:22.710258 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.710028 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xmfmq\"" Nov 04 02:32:22.710520 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.710507 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 02:32:22.711677 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.711565 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 02:32:22.711677 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.711568 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 02:32:22.711791 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.711685 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drzkl\"" Nov 04 02:32:22.730320 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.730287 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h9cmn"] Nov 04 02:32:22.730447 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.730419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.743595 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743574 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cz2p"] Nov 04 02:32:22.743698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743609 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzwxn"] Nov 04 02:32:22.743698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743633 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8xkb7"] Nov 04 02:32:22.743698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743656 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6"] Nov 04 02:32:22.743698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743676 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l2nk6"] Nov 04 02:32:22.743698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743695 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743715 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743734 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743754 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zp9vk"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743774 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nxqpl"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743793 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743818 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89rsm"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743837 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76598c4567-mhntk"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743858 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:32:22.743875 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743878 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn"] Nov 04 02:32:22.744247 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743907 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9cmn"] Nov 04 02:32:22.744247 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.743935 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:32:22.745566 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.745541 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7qp\" (UniqueName: \"kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp\") pod \"kas-connection-checker-685895c7c9-8xkb7\" (UID: \"64d03902-4d33-4bec-9725-53ec99ca7dbd\") " pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:22.745659 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.745622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfchl\" (UniqueName: \"kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl\") pod \"kas-connection-checker-685895c7c9-8cz2p\" (UID: \"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed\") " pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:22.752975 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.752941 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.752975 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.752949 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.752975 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.752964 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 02:32:22.753159 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.753014 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.753159 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.753029 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 02:32:22.753159 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.752966 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.754330 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.754312 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 02:32:22.760789 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.760773 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.760874 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.760804 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.782490 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.782460 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 02:32:22.782490 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.782467 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 02:32:22.783707 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.783784 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.783784 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.783784 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r2t\" (UniqueName: \"kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.783904 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.783943 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.783904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.784038 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.784098 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.784046 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.284029759 +0000 UTC m=+32.741894615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:22.784098 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptjw\" (UniqueName: \"kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.784224 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.784224 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.784224 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.784224 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.784250 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.784294 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.284281283 +0000 UTC m=+32.742146111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.784411 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.784614 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.784614 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvskz\" (UniqueName: \"kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.784614 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbjk\" (UniqueName: \"kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk\") pod \"kas-connection-checker-685895c7c9-bzwxn\" (UID: \"6e9a6a6d-33fc-45ea-9b03-af9724367475\") " pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:22.784614 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.784783 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.784819 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.784906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784888 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.784942 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784913 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.784973 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwfc\" (UniqueName: \"kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc\") pod \"volume-data-source-validator-58c9f5f46d-2chk6\" (UID: \"e5ee7cb7-c423-4701-b7ab-3ee603748499\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:22.785004 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.784973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.785037 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.785067 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwq9\" (UniqueName: \"kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.785116 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785146 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785245 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.785291 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785330 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.785330 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785407 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785407 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.785407 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785551 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.785464 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:22.785551 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.785484 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:22.785551 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.785529 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.28551278 +0000 UTC m=+32.743377611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.785695 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsvc\" (UniqueName: \"kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254jx\" (UniqueName: \"kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.785972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.785909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.786320 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.786284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:22.786435 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.786414 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:22.786435 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.786429 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:22.786555 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.786475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.286459615 +0000 UTC m=+32.744324458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:22.786555 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.786493 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.286483512 +0000 UTC m=+32.744348356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:22.786845 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.786566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.786988 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.786948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.787646 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.787620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.787908 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.787888 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.788113 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.788095 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.788344 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.788329 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.788616 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.788601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.796003 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.795985 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 02:32:22.800387 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800357 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 02:32:22.800387 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800359 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 02:32:22.800387 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800361 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 02:32:22.800387 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800388 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 02:32:22.800573 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800389 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6tfxk\"" Nov 04 02:32:22.800573 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.800518 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l8p2p\"" Nov 04 02:32:22.802619 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.802606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:22.822397 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.822370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:22.846053 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.845933 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbjk\" (UniqueName: \"kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk\") pod \"kas-connection-checker-685895c7c9-bzwxn\" (UID: \"6e9a6a6d-33fc-45ea-9b03-af9724367475\") " pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:22.846270 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.846243 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:22.846372 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.846341 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptjw\" (UniqueName: \"kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:22.846882 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.846862 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.846961 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.846909 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwfc\" (UniqueName: \"kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc\") pod \"volume-data-source-validator-58c9f5f46d-2chk6\" (UID: \"e5ee7cb7-c423-4701-b7ab-3ee603748499\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:22.847163 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.847144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:22.847459 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.847444 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwq9\" (UniqueName: \"kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.851428 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.851411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:22.886620 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.886620 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.886620 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttlh\" (UniqueName: \"kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r2t\" (UniqueName: \"kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886722 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9k6\" (UniqueName: \"kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6\") pod \"network-check-source-6f78d59f99-hm2jc\" (UID: \"3967e0a1-3d8a-4036-b369-34302fb7ee32\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvskz\" (UniqueName: \"kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.886906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.886992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsvc\" (UniqueName: \"kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254jx\" (UniqueName: \"kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnfj\" (UniqueName: \"kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.887540 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887527 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.888358 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887677 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.888358 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.887911 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:22.888358 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.887928 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:22.888358 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.887975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.888358 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.887995 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.387977062 +0000 UTC m=+32.845841898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:22.888731 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.888710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.889126 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.889101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.889445 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.889420 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.889656 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.889629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.889906 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.889881 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.890087 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.890154 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.890021 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:22.890291 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.890171 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.390154924 +0000 UTC m=+32.848019770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:22.890291 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890235 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.890398 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890296 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.890587 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.890927 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890901 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.891007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.890964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.891648 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.891609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.893355 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.893334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.916280 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.916245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:22.953407 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.952975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvskz\" (UniqueName: \"kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:22.956726 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.956113 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsvc\" (UniqueName: \"kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:22.956726 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.956577 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254jx\" (UniqueName: \"kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.957173 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.957144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r2t\" (UniqueName: \"kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:22.957602 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.957575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.957979 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.957932 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:22.968676 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.968636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:22.988663 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9k6\" (UniqueName: \"kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6\") pod \"network-check-source-6f78d59f99-hm2jc\" (UID: \"3967e0a1-3d8a-4036-b369-34302fb7ee32\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:22.988764 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.988798 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.989209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.989209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.989209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnfj\" (UniqueName: \"kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.989209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.988996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttlh\" (UniqueName: \"kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.989209 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.989160 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.989443 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.989229 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:22.989443 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:22.989295 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:23.48927466 +0000 UTC m=+32.947139491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:22.989644 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.989624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:22.991910 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.991879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:22.997639 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.997610 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:22.997732 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.997651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:22.998379 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:22.998139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:23.002701 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.002680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:23.038178 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.038123 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:23.050945 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.050907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9k6\" (UniqueName: \"kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6\") pod \"network-check-source-6f78d59f99-hm2jc\" (UID: \"3967e0a1-3d8a-4036-b369-34302fb7ee32\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:23.051401 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.051371 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-82bnw\"" Nov 04 02:32:23.052052 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.051869 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:23.052462 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.052441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttlh\" (UniqueName: \"kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:23.052615 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.052587 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnfj\" (UniqueName: \"kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:23.056868 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.056783 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8zwfm\"" Nov 04 02:32:23.056868 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.056783 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 02:32:23.057742 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.057106 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 02:32:23.058417 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.058388 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9fmkz\"" Nov 04 02:32:23.072401 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.072317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:23.133030 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.132977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:23.267724 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.267694 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6"] Nov 04 02:32:23.270497 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.270474 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cz2p"] Nov 04 02:32:23.271075 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.271050 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ee7cb7_c423_4701_b7ab_3ee603748499.slice/crio-60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4 WatchSource:0}: Error finding container 60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4: Status 404 returned error can't find the container with id 60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4 Nov 04 02:32:23.273242 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.273221 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe9ec5f_054e_4da3_b5d8_f2e4e32fd2ed.slice/crio-9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5 WatchSource:0}: Error finding container 9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5: Status 404 returned error can't find the container with id 9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5 Nov 04 02:32:23.292007 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.291974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:23.292134 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.292026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:23.292202 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.292156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:23.292263 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292206 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:23.292263 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.292233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:23.292360 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292281 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.292265828 +0000 UTC m=+33.750130670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:23.292360 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292202 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:23.292360 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292319 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.292300471 +0000 UTC m=+33.750165321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:23.292360 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292337 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:23.292360 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292351 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.292340289 +0000 UTC m=+33.750205115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:23.292558 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292381 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.29236734 +0000 UTC m=+33.750232184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:23.292558 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.292444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:23.292668 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292580 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:23.292668 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292595 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:23.292668 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.292646 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.292628819 +0000 UTC m=+33.750493663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:23.318430 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.318400 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8xkb7"] Nov 04 02:32:23.321654 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.321627 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d03902_4d33_4bec_9725_53ec99ca7dbd.slice/crio-132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d WatchSource:0}: Error finding container 132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d: Status 404 returned error can't find the container with id 132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d Nov 04 02:32:23.372326 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.372182 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2"] Nov 04 02:32:23.372326 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.372310 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2"] Nov 04 02:32:23.376859 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.376828 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06b3575_3fcd_4564_89b5_c7822dc36b59.slice/crio-220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1 WatchSource:0}: Error finding container 220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1: Status 404 returned error can't find the container with id 220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1 Nov 04 02:32:23.378311 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.378088 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bb9090_5872_40b1_ae31_b910dc6575fa.slice/crio-b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460 WatchSource:0}: Error finding container b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460: Status 404 returned error can't find the container with id b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460 Nov 04 02:32:23.393268 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.393233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:23.393409 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.393322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:23.393409 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.393390 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:23.393507 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.393412 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:23.393507 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.393459 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:23.393507 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.393471 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.393450903 +0000 UTC m=+33.851315750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:23.393507 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.393505 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.393492795 +0000 UTC m=+33.851357637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:23.415081 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.415050 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nxqpl"] Nov 04 02:32:23.424479 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.424449 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea891a3_ac68_4f11_add3_b42a1b657846.slice/crio-c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1 WatchSource:0}: Error finding container c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1: Status 404 returned error can't find the container with id c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1 Nov 04 02:32:23.452746 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.452719 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zp9vk"] Nov 04 02:32:23.455511 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.455481 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde579720_94f5_4cbd_b9f6_60e5b3d82647.slice/crio-8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9 WatchSource:0}: Error finding container 8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9: Status 404 returned error can't find the container with id 8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9 Nov 04 02:32:23.477827 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.477797 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc"] Nov 04 02:32:23.480975 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.480946 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3967e0a1_3d8a_4036_b369_34302fb7ee32.slice/crio-b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6 WatchSource:0}: Error finding container b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6: Status 404 returned error can't find the container with id b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6 Nov 04 02:32:23.484225 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.484182 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89rsm"] Nov 04 02:32:23.486811 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.486789 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdf7d9e_fc73_499b_9226_28e6b4c1d900.slice/crio-aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9 WatchSource:0}: Error finding container aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9: Status 404 returned error can't find the container with id aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9 Nov 04 02:32:23.494696 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.494669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:23.494788 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.494776 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:23.494844 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.494823 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.494811148 +0000 UTC m=+33.952675975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:23.534243 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.534203 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzwxn"] Nov 04 02:32:23.537716 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:23.537691 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9a6a6d_33fc_45ea_9b03_af9724367475.slice/crio-d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29 WatchSource:0}: Error finding container d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29: Status 404 returned error can't find the container with id d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29 Nov 04 02:32:23.999773 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.999698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:23.999942 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:23.999885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:24.000021 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:23.999986 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:24.000076 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.000037 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:56.000024896 +0000 UTC m=+65.457889742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:24.002216 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.002177 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:24.101285 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.101236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:24.104061 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.104032 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:24.182406 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.182369 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" event={"ID":"6e9a6a6d-33fc-45ea-9b03-af9724367475","Type":"ContainerStarted","Data":"f2abfe1e57eed61e2ff93a54b7fb04ac107c5715d252789e616d5d5dd617b94a"} Nov 04 02:32:24.182550 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.182431 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" event={"ID":"6e9a6a6d-33fc-45ea-9b03-af9724367475","Type":"ContainerStarted","Data":"d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29"} Nov 04 02:32:24.183629 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.183605 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89rsm" event={"ID":"8cdf7d9e-fc73-499b-9226-28e6b4c1d900","Type":"ContainerStarted","Data":"aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9"} Nov 04 02:32:24.184698 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.184678 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" event={"ID":"b1bb9090-5872-40b1-ae31-b910dc6575fa","Type":"ContainerStarted","Data":"b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460"} Nov 04 02:32:24.185791 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.185773 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" event={"ID":"3ea891a3-ac68-4f11-add3-b42a1b657846","Type":"ContainerStarted","Data":"c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1"} Nov 04 02:32:24.186876 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.186859 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" event={"ID":"3967e0a1-3d8a-4036-b369-34302fb7ee32","Type":"ContainerStarted","Data":"b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6"} Nov 04 02:32:24.187994 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.187977 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" event={"ID":"e5ee7cb7-c423-4701-b7ab-3ee603748499","Type":"ContainerStarted","Data":"60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4"} Nov 04 02:32:24.189294 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.189274 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" event={"ID":"64d03902-4d33-4bec-9725-53ec99ca7dbd","Type":"ContainerStarted","Data":"7920edaa6d2757b0bc3a2f935aa161a8af39ff462fe2c20a35fa7a2e00959b6a"} Nov 04 02:32:24.189349 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.189311 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" event={"ID":"64d03902-4d33-4bec-9725-53ec99ca7dbd","Type":"ContainerStarted","Data":"132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d"} Nov 04 02:32:24.190406 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.190388 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerStarted","Data":"8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9"} Nov 04 02:32:24.191428 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.191413 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" event={"ID":"a06b3575-3fcd-4564-89b5-c7822dc36b59","Type":"ContainerStarted","Data":"220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1"} Nov 04 02:32:24.192743 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.192724 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" event={"ID":"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed","Type":"ContainerStarted","Data":"f9bc682a10a6a783c09ac6a4856030f01c174e7e897225ba13b1b7146f3fd87b"} Nov 04 02:32:24.192823 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.192755 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" event={"ID":"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed","Type":"ContainerStarted","Data":"9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5"} Nov 04 02:32:24.226989 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.226957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:24.238930 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.238899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:24.303772 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.303671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:24.303832 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.303804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:24.304024 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.303899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:24.304024 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.304006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:24.304154 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.304097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:24.304409 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.304389 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:24.304468 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.304414 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:24.304520 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.304477 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.30445897 +0000 UTC m=+35.762323817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:24.304588 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.304546 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:24.304588 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.304582 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.304571761 +0000 UTC m=+35.762436590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:24.306670 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.306133 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:24.306670 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.306205 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.306170874 +0000 UTC m=+35.764035700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:24.306827 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.306810 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:24.306883 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.306864 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.306849572 +0000 UTC m=+35.764714410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:24.306976 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.306949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.306938398 +0000 UTC m=+35.764803224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:24.405115 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.405084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.405151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.405271 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.405293 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.405306 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.405356 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.405338043 +0000 UTC m=+35.863202884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:24.405623 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.405370 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.405364108 +0000 UTC m=+35.863228935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:24.506254 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.506143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:24.506408 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.506324 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:24.506408 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:24.506387 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:26.506370261 +0000 UTC m=+35.964235104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:24.578665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.578606 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" podStartSLOduration=417.578579345 podStartE2EDuration="6m57.578579345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:24.578259116 +0000 UTC m=+34.036123967" watchObservedRunningTime="2026-11-04 02:32:24.578579345 +0000 UTC m=+34.036444195" Nov 04 02:32:24.626328 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.626259 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65l2h"] Nov 04 02:32:24.629525 ip-10-0-133-209 kubenswrapper[2575]: W1104 02:32:24.629473 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e3e816_aa1d_4022_b606_0b4ecc1ee79e.slice/crio-0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa WatchSource:0}: Error finding container 0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa: Status 404 returned error can't find the container with id 0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa Nov 04 02:32:24.656239 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.655976 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xfw5"] Nov 04 02:32:24.692856 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.692803 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" podStartSLOduration=417.692786316 podStartE2EDuration="6m57.692786316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:24.691816245 +0000 UTC m=+34.149681094" watchObservedRunningTime="2026-11-04 02:32:24.692786316 +0000 UTC m=+34.150651159" Nov 04 02:32:24.806935 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:24.806815 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" podStartSLOduration=417.806798387 podStartE2EDuration="6m57.806798387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:24.806648218 +0000 UTC m=+34.264513066" watchObservedRunningTime="2026-11-04 02:32:24.806798387 +0000 UTC m=+34.264663237" Nov 04 02:32:25.201971 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:25.201927 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65l2h" event={"ID":"95e3e816-aa1d-4022-b606-0b4ecc1ee79e","Type":"ContainerStarted","Data":"0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa"} Nov 04 02:32:25.204972 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:25.204942 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xfw5" event={"ID":"59ccac4d-0b67-439b-a462-e11f04cdb14c","Type":"ContainerStarted","Data":"b7abfd27c191aeda89d07076c7b4870904854174ac4186f4989d44a9f7fc247a"} Nov 04 02:32:26.211874 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.211823 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" event={"ID":"e5ee7cb7-c423-4701-b7ab-3ee603748499","Type":"ContainerStarted","Data":"0cee0c2da78051f1d4f6137fc6001bb5018a84f01d419d2fa205932f97e050ad"} Nov 04 02:32:26.324290 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.324249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:26.324443 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.324373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:26.324443 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324402 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:26.324443 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.324427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:26.324539 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.324459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:26.324539 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324468 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.324453758 +0000 UTC m=+39.782318601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:26.324602 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.324536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:26.324602 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324540 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:26.324602 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324597 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:26.324682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324603 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.324591564 +0000 UTC m=+39.782456405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:26.324682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324610 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:26.324682 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324658 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.324642398 +0000 UTC m=+39.782507253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:26.324773 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324709 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:26.324773 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324738 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.324728266 +0000 UTC m=+39.782593123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:26.324894 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.324795 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.324785875 +0000 UTC m=+39.782650749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:26.327867 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.327819 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" podStartSLOduration=372.787343607 podStartE2EDuration="6m15.327806291s" totalImagesPullingTime="2.540462684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:26.327046054 +0000 UTC m=+35.784910903" watchObservedRunningTime="2026-11-04 02:32:26.327806291 +0000 UTC m=+35.785671137" Nov 04 02:32:26.425530 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.425490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:26.425712 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.425585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:26.425751 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.425740 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:26.425821 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.425810 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.425790943 +0000 UTC m=+39.883655790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:26.425899 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.425886 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:26.425933 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.425903 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:26.425961 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.425934 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.425924084 +0000 UTC m=+39.883788928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:26.526491 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:26.526278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:26.526924 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.526806 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:26.526924 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:26.526895 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.526873937 +0000 UTC m=+39.984738787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:28.220958 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:28.220925 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/0.log" Nov 04 02:32:28.221475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:28.220978 2575 generic.go:356] "Generic (PLEG): container finished" podID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerID="de7d13a54091deeded431e30aa7a7e24cfc711ac8c24fe6bbc8042b56f6f57eb" exitCode=255 Nov 04 02:32:28.221475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:28.221046 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerDied","Data":"de7d13a54091deeded431e30aa7a7e24cfc711ac8c24fe6bbc8042b56f6f57eb"} Nov 04 02:32:28.221475 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:28.221349 2575 scope.go:122] "RemoveContainer" containerID="de7d13a54091deeded431e30aa7a7e24cfc711ac8c24fe6bbc8042b56f6f57eb" Nov 04 02:32:29.226550 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.226468 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" event={"ID":"3ea891a3-ac68-4f11-add3-b42a1b657846","Type":"ContainerStarted","Data":"aac337142b53e93f28d60efccae61cfd7e147d7826f5049b25e1d17002bb992f"} Nov 04 02:32:29.228703 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.228677 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:29.229291 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.229268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/0.log" Nov 04 02:32:29.229405 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.229310 2575 generic.go:356] "Generic (PLEG): container finished" podID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" exitCode=255 Nov 04 02:32:29.229405 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.229348 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerDied","Data":"d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51"} Nov 04 02:32:29.229405 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.229399 2575 scope.go:122] "RemoveContainer" containerID="de7d13a54091deeded431e30aa7a7e24cfc711ac8c24fe6bbc8042b56f6f57eb" Nov 04 02:32:29.229793 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.229776 2575 scope.go:122] "RemoveContainer" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" Nov 04 02:32:29.230007 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:29.229980 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:29.473265 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:29.470911 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" podStartSLOduration=376.574413272 podStartE2EDuration="6m21.47088649s" totalImagesPullingTime="4.896473218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:29.34740394 +0000 UTC m=+38.805268789" watchObservedRunningTime="2026-11-04 02:32:29.47088649 +0000 UTC m=+38.928751345" Nov 04 02:32:30.060069 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.060026 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hl5s9"] Nov 04 02:32:30.103231 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.103170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.122304 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.122272 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hl5s9"] Nov 04 02:32:30.170665 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.170621 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 02:32:30.170853 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.170729 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2vt\"" Nov 04 02:32:30.236509 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.236484 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:30.236991 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.236841 2575 scope.go:122] "RemoveContainer" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" Nov 04 02:32:30.237093 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.237063 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:30.238602 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.238571 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89rsm" event={"ID":"8cdf7d9e-fc73-499b-9226-28e6b4c1d900","Type":"ContainerStarted","Data":"9bac7003e5fa78b9a52469b5b2b49978435a1c08f12447420abffec050ee185d"} Nov 04 02:32:30.240745 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.240708 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 02:32:30.275348 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.275307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.275535 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.275395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtptf\" (UniqueName: \"kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.275535 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.275500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.275655 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.275558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.275693 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.275644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.376495 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:30.376495 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376562 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376586 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376623 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtptf\" (UniqueName: \"kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376644 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376647 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.376628781 +0000 UTC m=+47.834493623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:30.376720 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376695 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376713 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.376680195 +0000 UTC m=+47.834545033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376755 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.376743407 +0000 UTC m=+47.834608239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376781 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.876768736 +0000 UTC m=+40.334633575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : configmap "kube-rbac-proxy" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376907 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.376900508 +0000 UTC m=+47.834765334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376949 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.376971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.376998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.876985547 +0000 UTC m=+40.334850402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.377014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.377169 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.377035 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:30.377800 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.377096 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.377080077 +0000 UTC m=+47.834944926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:30.377800 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.377099 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.377800 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.377496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.437670 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.437636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtptf\" (UniqueName: \"kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.468617 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.468554 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-89rsm" podStartSLOduration=2.774346925 podStartE2EDuration="8.468535092s" totalImagesPullingTime="5.694188167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:30.4671641 +0000 UTC m=+39.925028950" watchObservedRunningTime="2026-11-04 02:32:30.468535092 +0000 UTC m=+39.926399942" Nov 04 02:32:30.477897 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.477860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:30.478086 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.477954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:30.478163 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.478099 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:30.478163 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.478149 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.478137538 +0000 UTC m=+47.936002377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:30.478305 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.478228 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:30.478305 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.478239 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:30.478305 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.478279 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.478267608 +0000 UTC m=+47.936132449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:30.579515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.579472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:30.579684 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.579627 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:30.579751 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.579684 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.57966969 +0000 UTC m=+48.037534531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:30.882122 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.882085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.882311 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.882252 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.882311 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:30.882256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:30.882412 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.882321 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:31.882303385 +0000 UTC m=+41.340168240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.882412 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.882340 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 02:32:30.882412 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:30.882375 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:31.882367561 +0000 UTC m=+41.340232388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : configmap "kube-rbac-proxy" not found Nov 04 02:32:31.244498 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:31.244459 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" event={"ID":"3967e0a1-3d8a-4036-b369-34302fb7ee32","Type":"ContainerStarted","Data":"31671ef483771dc9994349da2d3668b7729114260f036577ffe82aef6bd789bc"} Nov 04 02:32:31.893515 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:31.893467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:31.893683 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:31.893630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:31.893683 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:31.893651 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:31.893780 ip-10-0-133-209 kubenswrapper[2575]: E1104 02:32:31.893731 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:33.893709835 +0000 UTC m=+43.351574681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:31.894229 ip-10-0-133-209 kubenswrapper[2575]: I1104 02:32:31.894179 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:32.943072 ip-10-0-133-209 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 02:32:32.955748 ip-10-0-133-209 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 02:32:32.955948 ip-10-0-133-209 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 02:32:32.956001 ip-10-0-133-209 systemd[1]: kubelet.service: Consumed 3.175s CPU time. Nov 04 02:32:32.965867 ip-10-0-133-209 systemd[1]: Starting Kubernetes Kubelet... Nov 04 02:32:33.063671 ip-10-0-133-209 kubenswrapper[4784]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.064280 ip-10-0-133-209 kubenswrapper[4784]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 02:32:33.064280 ip-10-0-133-209 kubenswrapper[4784]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.064280 ip-10-0-133-209 kubenswrapper[4784]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.066401 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066310 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.066401 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066402 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066406 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066409 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066412 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066416 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066418 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066421 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066424 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066426 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066429 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066432 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066434 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066437 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066439 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066442 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066445 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066448 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066450 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066453 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066455 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.066482 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066457 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066460 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066462 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066465 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066467 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066470 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066472 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066475 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066478 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066487 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066490 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066493 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066495 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066498 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066501 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066503 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066506 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066508 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066511 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066513 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.066984 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066516 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066519 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066521 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066524 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066526 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066529 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066531 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066534 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066537 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066540 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066542 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066545 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066548 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066551 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066553 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066556 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066558 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066561 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066564 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066566 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.067592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066569 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066571 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066574 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066576 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066579 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066582 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066584 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066589 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066593 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066596 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066599 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066601 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066604 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066606 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066609 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066611 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066614 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066617 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066619 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.068115 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066622 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066626 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066628 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066632 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066634 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066637 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066640 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066643 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066645 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066648 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066651 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066653 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066656 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066658 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066661 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066663 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066666 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066668 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.068613 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066671 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066676 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066679 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066682 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066686 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066689 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066691 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.066694 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067514 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067524 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067528 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067531 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067534 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067538 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067541 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067543 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067547 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067549 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067552 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.069076 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067555 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067558 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067560 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067563 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067566 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067569 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067571 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067574 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067576 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067580 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067583 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067585 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067588 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067591 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067593 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067596 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067599 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067601 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067604 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067607 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.069593 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067610 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067612 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067615 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067617 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067620 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067622 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067625 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067628 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067630 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067633 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067639 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067642 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067645 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067649 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067653 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067656 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067659 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067662 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067665 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067667 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.070118 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067670 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067672 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067675 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067677 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067680 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067682 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067686 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067688 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067691 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067693 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067695 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067698 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067700 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067703 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067705 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067708 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067710 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067713 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067715 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067718 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.070632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067720 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067722 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067725 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067728 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067734 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067737 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067740 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067742 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067745 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067747 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067750 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067753 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067755 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067758 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067760 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067763 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067766 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067768 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067771 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067773 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.071143 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067776 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067778 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067780 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067783 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067785 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067788 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067791 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067794 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067799 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067803 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067806 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067809 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067812 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067814 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.067817 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068377 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068388 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068391 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068394 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.071682 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068397 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068399 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068402 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068405 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068409 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068412 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068415 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068418 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068421 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068424 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068427 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068429 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068431 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068434 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068436 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068441 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068443 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068446 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068449 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.072156 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068451 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068454 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068457 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068459 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068462 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068464 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068467 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068469 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068472 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068474 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068484 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068487 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068490 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068493 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068495 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068498 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068500 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068503 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068505 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068508 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.072706 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068510 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068513 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068515 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068518 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068520 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068523 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068525 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068528 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068531 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068534 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068537 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068539 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068542 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068544 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068547 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068549 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068552 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068554 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068557 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.073229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068559 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068562 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068564 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068568 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068572 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068574 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068577 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068579 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068583 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068585 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068588 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068590 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068593 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068595 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068598 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068600 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068603 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068606 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068611 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.073701 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068613 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068616 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068621 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068623 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068626 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068629 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068631 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068634 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068636 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068639 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068642 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068644 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068647 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068649 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068652 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068654 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068658 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068660 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068664 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068666 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.074229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068669 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068671 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068674 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068676 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.068680 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068779 4784 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068795 4784 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068802 4784 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068811 4784 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068816 4784 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068819 4784 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068823 4784 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068828 4784 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068831 4784 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068836 4784 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068840 4784 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068843 4784 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068846 4784 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068849 4784 flags.go:64] FLAG: --cgroup-root="" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068852 4784 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068855 4784 flags.go:64] FLAG: --client-ca-file="" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068858 4784 flags.go:64] FLAG: --cloud-provider="external" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068861 4784 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068865 4784 flags.go:64] FLAG: --cluster-domain="" Nov 04 02:32:33.074722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068868 4784 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068871 4784 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068874 4784 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068878 4784 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068882 4784 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068885 4784 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068888 4784 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068892 4784 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068895 4784 flags.go:64] FLAG: --contention-profiling="false" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068898 4784 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068901 4784 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068905 4784 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068907 4784 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068912 4784 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068915 4784 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068918 4784 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068920 4784 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068924 4784 flags.go:64] FLAG: --enable-server="true" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068927 4784 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068932 4784 flags.go:64] FLAG: --event-burst="100" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068935 4784 flags.go:64] FLAG: --event-qps="50" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068938 4784 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068943 4784 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068946 4784 flags.go:64] FLAG: --eviction-hard="" Nov 04 02:32:33.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068950 4784 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068953 4784 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068957 4784 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068960 4784 flags.go:64] FLAG: --eviction-soft="" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068963 4784 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068966 4784 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068969 4784 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068972 4784 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068976 4784 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068978 4784 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068981 4784 flags.go:64] FLAG: --feature-gates="" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068985 4784 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068989 4784 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068992 4784 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068995 4784 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.068998 4784 flags.go:64] FLAG: --healthz-port="10248" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069001 4784 flags.go:64] FLAG: --help="false" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069005 4784 flags.go:64] FLAG: --hostname-override="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069008 4784 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069011 4784 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069014 4784 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069017 4784 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069020 4784 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 02:32:33.076035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069023 4784 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069026 4784 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069029 4784 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069031 4784 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069034 4784 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069038 4784 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069041 4784 flags.go:64] FLAG: --kube-reserved="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069045 4784 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069047 4784 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069051 4784 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069053 4784 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069056 4784 flags.go:64] FLAG: --lock-file="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069059 4784 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069062 4784 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069065 4784 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069075 4784 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069079 4784 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069082 4784 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069085 4784 flags.go:64] FLAG: --logging-format="text" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069088 4784 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069092 4784 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069095 4784 flags.go:64] FLAG: --manifest-url="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069098 4784 flags.go:64] FLAG: --manifest-url-header="" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069102 4784 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069105 4784 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 02:32:33.076631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069110 4784 flags.go:64] FLAG: --max-pods="110" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069113 4784 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069117 4784 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069120 4784 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069123 4784 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069126 4784 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069129 4784 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069132 4784 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069140 4784 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069143 4784 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069146 4784 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069150 4784 flags.go:64] FLAG: --pod-cidr="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069153 4784 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069156 4784 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069161 4784 flags.go:64] FLAG: --pods-per-core="0" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069164 4784 flags.go:64] FLAG: --port="10250" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069167 4784 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069170 4784 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d2b6b6e3a25a8564" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069173 4784 flags.go:64] FLAG: --qos-reserved="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069176 4784 flags.go:64] FLAG: --read-only-port="10255" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069179 4784 flags.go:64] FLAG: --register-node="true" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069182 4784 flags.go:64] FLAG: --register-with-taints="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069200 4784 flags.go:64] FLAG: --registry-burst="10" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069203 4784 flags.go:64] FLAG: --registry-qps="5" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069205 4784 flags.go:64] FLAG: --reserved-cpus="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069208 4784 flags.go:64] FLAG: --reserved-memory="" Nov 04 02:32:33.077364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069213 4784 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069216 4784 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069218 4784 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069221 4784 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069224 4784 flags.go:64] FLAG: --runonce="false" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069227 4784 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069230 4784 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069233 4784 flags.go:64] FLAG: --seccomp-default="false" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069237 4784 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069240 4784 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069243 4784 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069246 4784 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069250 4784 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069252 4784 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069255 4784 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069258 4784 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069261 4784 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069264 4784 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069267 4784 flags.go:64] FLAG: --system-cgroups="" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069270 4784 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069278 4784 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069281 4784 flags.go:64] FLAG: --tls-cert-file="" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069283 4784 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069287 4784 flags.go:64] FLAG: --tls-min-version="" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069290 4784 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 02:32:33.078024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069293 4784 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069296 4784 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069299 4784 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069302 4784 flags.go:64] FLAG: --v="2" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069306 4784 flags.go:64] FLAG: --version="false" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069310 4784 flags.go:64] FLAG: --vmodule="" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069314 4784 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069318 4784 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069471 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069474 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069477 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069480 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069482 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069485 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069487 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069490 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069493 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069496 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069498 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069501 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069504 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.078655 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069506 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069508 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069511 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069513 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069516 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069518 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069522 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069525 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069527 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069530 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069532 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069535 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069538 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069541 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069543 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069546 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069548 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069551 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069554 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.079360 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069556 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069559 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069561 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069564 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069566 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069569 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069571 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069574 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069576 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069579 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069582 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069584 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069587 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069589 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069592 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069594 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069597 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069600 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069602 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069606 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.080106 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069609 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069612 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069614 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069617 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069619 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069622 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069624 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069628 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069631 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069633 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069636 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069640 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069644 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069647 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069650 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069653 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069656 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069658 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069661 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069663 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.080824 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069666 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069669 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069672 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069675 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069677 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069680 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069682 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069685 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069687 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069690 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069692 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069697 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069699 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069702 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069704 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069707 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069709 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069712 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069716 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069722 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.081436 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069726 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069730 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069737 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069743 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069747 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069750 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069753 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069756 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069759 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069762 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069764 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069767 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069770 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069773 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.069801 4784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.081996 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069850 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069855 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069858 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069862 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069865 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069867 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069870 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069872 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069876 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069879 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069882 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069884 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069887 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069890 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069893 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069896 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069898 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069902 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069905 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069907 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.082451 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069910 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069912 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069915 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069918 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069920 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069923 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069925 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069928 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069930 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069933 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069936 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069938 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069941 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069944 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069946 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069951 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069954 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069957 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069959 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069962 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.083493 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069966 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069968 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069971 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069974 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069976 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069979 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069982 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069984 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069987 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069992 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069995 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.069998 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070000 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070003 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070005 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070008 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070010 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070013 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070015 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070018 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.084243 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070020 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070023 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070025 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070028 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070030 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070033 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070037 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070042 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070045 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070047 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070050 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070052 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070056 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070059 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070061 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070064 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070066 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070069 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070072 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.084831 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070074 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070077 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070080 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070083 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070085 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070088 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070091 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070093 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070096 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070098 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070101 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070103 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070106 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070108 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070110 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070113 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070115 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070118 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070121 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.085372 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070123 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070127 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070130 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070132 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070135 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070138 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070142 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.070144 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.074329 4784 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.074348 4784 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.074366 4784 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.074371 4784 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074440 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074446 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074451 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074454 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074456 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074459 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074462 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.085834 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074464 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074467 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074469 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074472 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074474 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074477 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074480 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074483 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074485 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074488 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074491 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074493 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074495 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074498 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074508 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074511 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074514 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074517 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074519 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074522 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.086314 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074524 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074527 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074530 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074532 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074535 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074537 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074540 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074543 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074545 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074548 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074550 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074553 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074556 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074558 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074561 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074563 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074566 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074568 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074571 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.086814 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074573 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074577 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074581 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074584 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074587 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074590 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074593 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074596 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074604 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074606 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074610 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074612 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074615 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074618 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074620 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074623 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074625 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074628 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074631 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074633 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.087289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074635 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074638 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074641 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074644 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074646 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074649 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074651 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074654 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074658 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074661 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074665 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074667 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074670 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074673 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074675 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074678 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074680 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074683 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.087775 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074685 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074689 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074691 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074699 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074702 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074705 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074707 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074710 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074713 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074715 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074718 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074721 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074724 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074727 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074729 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074732 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074734 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074738 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074740 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074743 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.088335 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074745 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074748 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.074777 4784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074969 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074974 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074976 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074979 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074982 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074985 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074987 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074989 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074992 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074994 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.074997 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075000 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075003 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.088816 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075011 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075014 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075016 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075019 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075022 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075024 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075027 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075031 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075034 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075037 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075039 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075042 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075044 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075047 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075049 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075051 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075054 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075056 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075059 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.089220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075061 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075064 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075066 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075069 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075071 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075073 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075076 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075078 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075081 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075083 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075086 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075089 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075091 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075095 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075103 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075106 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075108 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075111 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075113 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.089687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075116 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075118 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075121 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075123 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075126 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075129 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075131 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075133 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075136 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075138 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075142 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075146 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075150 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075152 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075155 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075158 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075160 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075163 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075165 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.090148 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075167 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075170 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075172 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075175 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075178 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075180 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075183 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075200 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075204 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075223 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075226 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075229 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075231 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075234 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075237 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075239 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075242 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075245 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075247 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075250 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.090629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075258 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075261 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075263 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075265 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075268 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075270 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075273 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075276 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075278 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075281 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075283 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075286 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075288 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075291 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075293 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075296 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.091108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.075324 4784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075375 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075380 4784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075384 4784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075387 4784 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075390 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075400 4784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075404 4784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075408 4784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075411 4784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075414 4784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075416 4784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075419 4784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075422 4784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075424 4784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075427 4784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075429 4784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075431 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075434 4784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.091522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075436 4784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075439 4784 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075441 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075444 4784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075446 4784 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075449 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075451 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075454 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075456 4784 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075459 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075461 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075463 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075466 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075469 4784 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075472 4784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075474 4784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075477 4784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075479 4784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075482 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075484 4784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.092067 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075492 4784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075495 4784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075498 4784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075500 4784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075503 4784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075505 4784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075508 4784 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075510 4784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075513 4784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075515 4784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075518 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075520 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075522 4784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075525 4784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075527 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075530 4784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075532 4784 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075534 4784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075537 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075539 4784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.092734 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075542 4784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075544 4784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075546 4784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075549 4784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075551 4784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075554 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075559 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075561 4784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075564 4784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075566 4784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075569 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075573 4784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075576 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075585 4784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075588 4784 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075590 4784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075593 4784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075595 4784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075598 4784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075600 4784 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.093236 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075602 4784 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075605 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075608 4784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075610 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075613 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075615 4784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075618 4784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075620 4784 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075622 4784 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075625 4784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075627 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075630 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075632 4784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075635 4784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075637 4784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075640 4784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075642 4784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075645 4784 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075650 4784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.093825 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075652 4784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075655 4784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075657 4784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075660 4784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075662 4784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075665 4784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075667 4784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075677 4784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:33.075681 4784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.075737 4784 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.076868 4784 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.076935 4784 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.077113 4784 server.go:1014] "Starting client certificate rotation" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.077284 4784 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.077379 4784 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 02:26:51 +0000 UTC" deadline="2028-07-10 21:15:59.808397671 +0000 UTC" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.077441 4784 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14754h43m26.730958432s" Nov 04 02:32:33.094322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.077717 4784 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:32:33.094728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.079320 4784 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:32:33.094728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.081933 4784 log.go:25] "Validated CRI v1 runtime API" Nov 04 02:32:33.094728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.083952 4784 log.go:25] "Validated CRI v1 image API" Nov 04 02:32:33.094728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.084697 4784 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 02:32:33.094728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.086554 4784 fs.go:135] Filesystem UUIDs: map[43e7832b-81cf-49b6-b10b-9d73211cdf30:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d323fe12-2cfb-4cc7-a99b-7664220f4256:/dev/nvme0n1p3] Nov 04 02:32:33.094873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.086572 4784 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7abfd27c191aeda89d07076c7b4870904854174ac4186f4989d44a9f7fc247a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7abfd27c191aeda89d07076c7b4870904854174ac4186f4989d44a9f7fc247a/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0009777c-b015-471c-8d88-72d341de5778/volumes/kubernetes.io~projected/kube-api-access-kvskz:{mountpoint:/var/lib/kubelet/pods/0009777c-b015-471c-8d88-72d341de5778/volumes/kubernetes.io~projected/kube-api-access-kvskz major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/tmp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~projected/kube-api-access-566wr:{mountpoint:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~projected/kube-api-access-566wr major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/220ee58a-5ca9-4f09-a317-e5b66524efb0/volumes/kubernetes.io~projected/kube-api-access-dxsj8:{mountpoint:/var/lib/kubelet/pods/220ee58a-5ca9-4f09-a317-e5b66524efb0/volumes/kubernetes.io~projected/kube-api-access-dxsj8 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3967e0a1-3d8a-4036-b369-34302fb7ee32/volumes/kubernetes.io~projected/kube-api-access-4n9k6:{mountpoint:/var/lib/kubelet/pods/3967e0a1-3d8a-4036-b369-34302fb7ee32/volumes/kubernetes.io~projected/kube-api-access-4n9k6 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~projected/kube-api-access-v9plm:{mountpoint:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~projected/kube-api-access-v9plm major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~projected/kube-api-access-l8r2t:{mountpoint:/var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~projected/kube-api-access-l8r2t major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584afc7e-72fd-4893-a77c-519e4b0b44ad/volumes/kubernetes.io~projected/kube-api-access-b2pvs:{mountpoint:/var/lib/kubelet/pods/584afc7e-72fd-4893-a77c-519e4b0b44ad/volumes/kubernetes.io~projected/kube-api-access-b2pvs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59ccac4d-0b67-439b-a462-e11f04cdb14c/volumes/kubernetes.io~projected/kube-api-access-8g9f9:{mountpoint:/var/lib/kubelet/pods/59ccac4d-0b67-439b-a462-e11f04cdb14c/volumes/kubernetes.io~projected/kube-api-access-8g9f9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d03902-4d33-4bec-9725-53ec99ca7dbd/volumes/kubernetes.io~projected/kube-api-access-lx7qp:{mountpoint:/var/lib/kubelet/pods/64d03902-4d33-4bec-9725-53ec99ca7dbd/volumes/kubernetes.io~projected/kube-api-access-lx7qp major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e9a6a6d-33fc-45ea-9b03-af9724367475/volumes/kubernetes.io~projected/kube-api-access-7xbjk:{mountpoint:/var/lib/kubelet/pods/6e9a6a6d-33fc-45ea-9b03-af9724367475/volumes/kubernetes.io~projected/kube-api-access-7xbjk major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78af83a8-adea-4abe-962e-c95203bdcde2/volumes/kubernetes.io~projected/kube-api-access-vwdf7:{mountpoint:/var/lib/kubelet/pods/78af83a8-adea-4abe-962e-c95203bdcde2/volumes/kubernetes.io~projected/kube-api-access-vwdf7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/kube-api-access-gnslq:{mountpoint:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/kube-api-access-gnslq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82a5e4c2-b127-4bfe-9a5b-943e48410287/volumes/kubernetes.io~projected/kube-api-access-wrspf:{mountpoint:/var/lib/kubelet/pods/82a5e4c2-b127-4bfe-9a5b-943e48410287/volumes/kubernetes.io~projected/kube-api-access-wrspf major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/864630d2-d25b-4d12-aa4a-c9a60f5a13f4/volumes/kubernetes.io~projected/kube-api-access-sg9vn:{mountpoint:/var/lib/kubelet/pods/864630d2-d25b-4d12-aa4a-c9a60f5a13f4/volumes/kubernetes.io~projected/kube-api-access-sg9vn major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~projected/kube-api-access-xttlh:{mountpoint:/var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~projected/kube-api-access-xttlh major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~secret/cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed/volumes/kubernetes.io~projected/kube-api-access-nfchl:{mountpoint:/var/lib/kubelet/pods/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed/volumes/kubernetes.io~projected/kube-api-access-nfchl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95e3e816-aa1d-4022-b606-0b4ecc1ee79e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/95e3e816-aa1d-4022-b606-0b4ecc1ee79e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~projected/kube-api-access-8xwq9:{mountpoint:/var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~projected/kube-api-access-8xwq9 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5d950c5-819d-4cc5-aa08-7f80bf099eb1/volumes/kubernetes.io~projected/kube-api-access-xqwhl:{mountpoint:/var/lib/kubelet/pods/a5d950c5-819d-4cc5-aa08-7f80bf099eb1/volumes/kubernetes.io~projected/kube-api-access-xqwhl major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~projected/kube-api-access-254jx:{mountpoint:/var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~projected/kube-api-access-254jx major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2376a8d-13b4-490b-9f66-b55cf5f3a51f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b2376a8d-13b4-490b-9f66-b55cf5f3a51f/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a19051-1ac9-4caf-82df-5a004c49db68/volumes/kubernetes.io~projected/kube-api-access-rtptf:{mountpoint:/var/lib/kubelet/pods/c1a19051-1ac9-4caf-82df-5a004c49db68/volumes/kubernetes.io~projected/kube-api-access-rtptf major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f/volumes/kubernetes.io~projected/kube-api-access-t2bvr:{mountpoint:/var/lib/kubelet/pods/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f/volumes/kubernetes.io~projected/kube-api-access-t2bvr major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0ca244-6231-4058-9051-8aa91ae0ce73/volumes/kubernetes.io~projected/kube-api-access-g8tmv:{mountpoint:/var/lib/kubelet/pods/cb0ca244-6231-4058-9051-8aa91ae0ce73/volumes/kubernetes.io~projected/kube-api-access-g8tmv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~projected/kube-api-access-7wsvc:{mountpoint:/var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~projected/kube-api-access-7wsvc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5ee7cb7-c423-4701-b7ab-3ee603748499/volumes/kubernetes.io~projected/kube-api-access-ldwfc:{mountpoint:/var/lib/kubelet/pods/e5ee7cb7-c423-4701-b7ab-3ee603748499/volumes/kubernetes.io~projected/kube-api-access-ldwfc major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ece428ad-e7c3-4744-925a-036290ed726e/volumes/kubernetes.io~projected/kube-api-access-lgnfj:{mountpoint:/var/lib/kubelet/pods/ece428ad-e7c3-4744-925a-036290ed726e/volumes/kubernetes.io~projected/kube-api-access-lgnfj major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/kube-api-access-ctdbg:{mountpoint:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/kube-api-access-ctdbg major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~projected/kube-api-access-4ptjw:{mountpoint:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~projected/kube-api-access-4ptjw major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3da47b01c8dd47af3cc3b0d96db576c14ea4461fe325dd052d5ae08ef16988c7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/aa3b6718a46972e5d3c3058de242254664f8583872b2094ac992bac94eded340/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d9ebdea5bf20c6f0fe35b63247a27b788a7bfe65eb95ae71641b62f14f9a0219/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/33e3ae98dc3066af089d6856be98442c7dac0177f78271255d2f6fbd37604648/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/722090eeb92f313b72e6c13cdb975108b19400a46f89577fadcaa9fd0ddde00b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/dd25e2ed5f0f49b2796d792ac86ebc1211d21f5182c42a7e461405d1bfaf3ebb/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b8a53fcf0e66287a92e81b58b46b9fc72dd6981e1e39167e609091ce68313187/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30813d2fb8bdafb54fccc61c3898146a6fc432d5a3bdb38e966e0716dac18742/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ca6f00142302eca32ff18eae64435115e2992d79172cf0df5b78809df9e8420a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/462d63b389207ea075afe398feaa6d569b039147c89c4f1aa27d638dc54568b5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/57d985f413157d1dc81cfb62ad8db5bb55b65d214d475aa3ce06c898f5c6a959/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/d5eab5de26d134679cce51f54fc00aea1e213608c4a493063f2e6d8e60a05e94/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/96da5facf1bbcf18a12461a38a30c2e5a1b347b88a56dbb8ea8520a866a283b1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/70115b611ec72e57eb2cdc29e71c93ef28308e5ecb785b6b540d78e86003945f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7b11d51ba4269406cf99b371b049d038e32f047b23775f47cc440f19a4fa5aeb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8deaa089f4edb4fbe5c92feb89686de42332a2043f00bac5aed27f420a3c6c5f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fcb53f3b6f1ee2c155bb2d1ab66fcc3fe49186ca134a979e07e29bcb8973da7c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2d283cde7ee23b31ca09beac174a4b4b65c37352cad4aca907146f9325476e70/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4111d87d6dad008bf9ad8b050f73a0094b4795b1a4564171b6c684a15bddf9b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/89e9280ab6f99365f77643de4a90e257a1b40cf9c6a549c2e64eb26ee3f31583/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5cbed7ac8f3bf4ca8c50bab69af9fb84b009431b795e03ccf2fb877a417c8bb4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/51594c518259ef9ebe08fa7ceffc523369b137999147c3800454e456c5ac3351/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/92040f3849d7a5f6519436bde3b56a69bc877f98a6d8da540b7f3ff25665841f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/df29bbf9425c36998367b6658dddbe2143ad92a170f2857635e5621812609d99/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/946d58452a94969088ac633e366f2297b50138248c266fed827e21e99c3228a2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/69fd4aff8e622c6f597e4d3dacc74f29144723f93f0aff81afbdd9c183c70b1d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d03bf39a8531a18b00257f98aafed4081b1dd26f414bbf7b7cd90fca8c7d2d0a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/bdfc5f7dd2020b03fc24afde0b08963ac61a30a8c77b92c19cbea085527ae080/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/c03bdd6a891b7f1ba20a1a5357eefa841e10f067227ffbc7da76f83d371fb58c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/4959b18d79bd0bfcc48086b8036443730e5bd7df2f5f4c7b3238f28ae9375dc0/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/824f696dc5c609b713e6bd32c4bd2c6f965b0c958d7d14e7e4fe08dbfce5ed3b/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b86a662e106fd3dbbf3d4c34d529b26e0f8d6be057f535c59371fc0fcdef16a8/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/0710eb6e26a598daad4222a700b69d8e755e69d69a35899410dca231865a320c/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/314c14b8f7875ff104b9a983b35dacec84fadbc6835db94059ba83f622190baf/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/4c54e737837f5f24f0a74ea54aa24c3bbcc702ae3ea94a01a67a6ba59898c8c5/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/c307e4206506be89af5d1bcf3bd7fd5834a27dd5bb9bfdd8cb7109c97c7789a0/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/7a1fd3102f3b06fe763efdf1323fb9601d06e2e3652cdc78336a64ff2a696639/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/942361a2751021c821645b36f44ac46e842c701d75fca508921e95734a789d5b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/311b1036d231452801cec1e54a8da64458b60d50e3692d60b04df47b73dc1379/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/21987265b4c93fbd0cfa091724a343af1d1b410d32354982c36f85429a74b4f3/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/079a1c09cf15f2cda805553deb76505fa81420b72fa73bf293a6ed96e115df12/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ad2e264f863055b482865c4faf6e9cb2172fe9f4200152806c684b153015931d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a258a6272fcfc1898aabf21cccc3e01e7f14c57c6be3936fc9b0715d1e683538/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/989490c4f869540c4b47075e8bc8f907eae13546df28191ca8dcba963f1ef141/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4ce54cba334230ce1372c084a438bcea05bbfbd28cb2688b0e06a4770d0e982b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5db1a448c97d8a1d8d8b192c46adaa24e7ae2e1d2da0c0d2f744aa651e5b506f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e9dab83513e7b9ccb76a43f3e323e870bb89fc04a32d22de002c9ad2ba3eddfe/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/592eb26dbf9e4d190b4336ef9c00362bcc2cfd19ad0dbe994d23552bbc4d6772/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/1f32f06852da066a03b46b3ee32efd90699638072581295ac14aa0486f636c5d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4c1df9b163e88e94afc4216b16b609799e3bf668ee7313f9b2c02245b9357274/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b82642a841545d09cf7cd81299f9d4a9aee69078ec4102c9be21094fd25b3d4e/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 04 02:32:33.095090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.094638 4784 manager.go:217] Machine: {Timestamp:2026-11-04 02:32:33.094129854 +0000 UTC m=+0.060630837 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127278 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269919f03f05cdeaf4c668bb5a23fb SystemUUID:ec269919-f03f-05cd-eaf4-c668bb5a23fb BootID:87d03c5d-3e5a-449e-86c1-f3f879f0fb29 Filesystems:[{Device:/var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~projected/kube-api-access-8xwq9 DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/kube-api-access-ctdbg DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3967e0a1-3d8a-4036-b369-34302fb7ee32/volumes/kubernetes.io~projected/kube-api-access-4n9k6 DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e9a6a6d-33fc-45ea-9b03-af9724367475/volumes/kubernetes.io~projected/kube-api-access-7xbjk DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1bb9090-5872-40b1-ae31-b910dc6575fa/volumes/kubernetes.io~projected/kube-api-access-254jx DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/132a970843e03e327178202535e17af1453331e54dd7eedd522ac5313cdf6e9d/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7abfd27c191aeda89d07076c7b4870904854174ac4186f4989d44a9f7fc247a/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0a9c092d5f50f491133ec004f1b97c4c0252b8bd9654f6a2fd0eb6e420d744d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82a5e4c2-b127-4bfe-9a5b-943e48410287/volumes/kubernetes.io~projected/kube-api-access-wrspf DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78af83a8-adea-4abe-962e-c95203bdcde2/volumes/kubernetes.io~projected/kube-api-access-vwdf7 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/208beb154fa5e18c94ffb10c3b0ff9433a6c2c69c00902123f36ad427af91faa/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b6466a281983448d082d6878557a5b9c7fe49a3c6ddf885f07f18cc6b8339f9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d89ead298a201e9f965b0f742692dbbdd12a5d018ff01a37373ad32b1b5145a/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~projected/kube-api-access-gnslq DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d73e42e8d49669b2e39ac83ba8bbcf942c8737ab353ac74a8a26138542d07a29/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95e3e816-aa1d-4022-b606-0b4ecc1ee79e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0041bf28595111980f1a6dd11c7452462a602c3bed0890bfcc295b25b91141aa/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60e9fd3245da9c91fc76b9b402a74947f69d38e35ebabbc0f4323ab465c6b5e4/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f/volumes/kubernetes.io~projected/kube-api-access-t2bvr DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdf7d9e-fc73-499b-9226-28e6b4c1d900/volumes/kubernetes.io~projected/kube-api-access-xttlh DeviceMajor:0 DeviceMinor:238 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1ed0cc10f5c8c937909c5770447acb65aed882382d15289abda44e34c074ea6/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5f3660e5c22797efd826e4af9b6ac1be8f140e77b7f27ec36922d5c2df62ad7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~projected/kube-api-access-v9plm DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/220ee58a-5ca9-4f09-a317-e5b66524efb0/volumes/kubernetes.io~projected/kube-api-access-dxsj8 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9242266640cb7d99ce08761d5183128a5b50712ffc9c24a3070b1b7a489bd9d5/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b72a61d9f6caeaa9a51b630c600d16994a2a184faa978b8771b3445afec13460/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c95357cb53a915cc7f805edb085d03a5ec7bfdf82279ebba3f420ac2760cbeb1/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a715d2cfcda629cf06e8fc217ca09832958ea24ecfd2cc11d6a02639cdf0138/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b2376a8d-13b4-490b-9f66-b55cf5f3a51f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06b3575-3fcd-4564-89b5-c7822dc36b59/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0abdefaf-ed85-4c17-bfe7-11b0aec18b44/volumes/kubernetes.io~projected/kube-api-access-566wr DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aece075a8db5aed7fba17c9d6d12c01265a1fd31d224f745e85f2833aab580e9/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a849d23ccda3ede3c472b665381795dbb54c731830b71373effe8f75b68efa59/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~projected/kube-api-access-7wsvc DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64d03902-4d33-4bec-9725-53ec99ca7dbd/volumes/kubernetes.io~projected/kube-api-access-lx7qp DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/220e3ee4bfa0f90eceecc6d98c2b38b6c97e736c58bb09427b2cb8e1655e64e1/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ece428ad-e7c3-4744-925a-036290ed726e/volumes/kubernetes.io~projected/kube-api-access-lgnfj DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/59ccac4d-0b67-439b-a462-e11f04cdb14c/volumes/kubernetes.io~projected/kube-api-access-8g9f9 DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/864630d2-d25b-4d12-aa4a-c9a60f5a13f4/volumes/kubernetes.io~projected/kube-api-access-sg9vn DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5ee7cb7-c423-4701-b7ab-3ee603748499/volumes/kubernetes.io~projected/kube-api-access-ldwfc DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de579720-94f5-4cbd-b9f6-60e5b3d82647/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a19051-1ac9-4caf-82df-5a004c49db68/volumes/kubernetes.io~projected/kube-api-access-rtptf DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/399e71a6-e15f-4aea-b7fc-25f45d6c182e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5d950c5-819d-4cc5-aa08-7f80bf099eb1/volumes/kubernetes.io~projected/kube-api-access-xqwhl DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0009777c-b015-471c-8d88-72d341de5778/volumes/kubernetes.io~projected/kube-api-access-kvskz DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/584afc7e-72fd-4893-a77c-519e4b0b44ad/volumes/kubernetes.io~projected/kube-api-access-b2pvs DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed/volumes/kubernetes.io~projected/kube-api-access-nfchl DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ea891a3-ac68-4f11-add3-b42a1b657846/volumes/kubernetes.io~projected/kube-api-access-l8r2t DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0ca244-6231-4058-9051-8aa91ae0ce73/volumes/kubernetes.io~projected/kube-api-access-g8tmv DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3823b7600324efdfe35f9dab458a08c9137763e659442c07e246b7bdf881e0a9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a11907b9c0fedb58a565967c9ffa122c625af5b21a109f7e850eb80e69778abf/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7e78e94-cc94-496f-ac33-f4e708709d9a/volumes/kubernetes.io~projected/kube-api-access-4ptjw DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0041bf285951119 MacAddress:3e:9b:2b:1c:0c:4e Speed:10000 Mtu:8901} {Name:132a970843e03e3 MacAddress:3a:6c:21:28:16:f2 Speed:10000 Mtu:8901} {Name:220e3ee4bfa0f90 MacAddress:e6:c9:e8:f4:f0:93 Speed:10000 Mtu:8901} {Name:60e9fd3245da9c9 MacAddress:ca:cb:02:a7:8d:57 Speed:10000 Mtu:8901} {Name:8b6466a28198344 MacAddress:0a:e6:56:c1:8c:e6 Speed:10000 Mtu:8901} {Name:9242266640cb7d9 MacAddress:2a:e0:22:8c:ca:03 Speed:10000 Mtu:8901} {Name:aece075a8db5aed MacAddress:8e:e0:8b:6f:3f:d0 Speed:10000 Mtu:8901} {Name:b1ed0cc10f5c8c9 MacAddress:22:f4:88:61:d4:52 Speed:10000 Mtu:8901} {Name:b72a61d9f6caeaa MacAddress:76:cf:57:4f:12:29 Speed:10000 Mtu:8901} {Name:b7abfd27c191aed MacAddress:46:8b:23:12:2c:d1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:2c:49:00:d0:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:86:45:97:dd:d6:1d Speed:0 Mtu:8901} {Name:c95357cb53a915c MacAddress:0e:47:d5:d3:ad:39 Speed:10000 Mtu:8901} {Name:d73e42e8d49669b MacAddress:6a:90:4d:e8:e2:5d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:2c:49:00:d0:4d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6a:8d:4a:4b:ba:1b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:95:c9:b0:03:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 02:32:33.095090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.094993 4784 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 02:32:33.095090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095041 4784 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 02:32:33.095433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095407 4784 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 02:32:33.095590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095432 4784 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-209.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 02:32:33.095635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095592 4784 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 02:32:33.095635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095600 4784 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 02:32:33.095635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095611 4784 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:32:33.095635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095629 4784 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:32:33.095739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095649 4784 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 02:32:33.095864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095854 4784 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 02:32:33.095979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.095971 4784 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 02:32:33.096017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096009 4784 kubelet.go:493] "Attempting to sync node with API server" Nov 04 02:32:33.096045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096024 4784 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 02:32:33.096045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096034 4784 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 02:32:33.096045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096044 4784 kubelet.go:405] "Adding apiserver pod source" Nov 04 02:32:33.096137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096052 4784 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 02:32:33.096858 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096842 4784 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 02:32:33.096858 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.096855 4784 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:32:33.099095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.099072 4784 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 02:32:33.108944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.108918 4784 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 02:32:33.108944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.108939 4784 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:32:33.109112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109061 4784 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 02:32:33.109302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109290 4784 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 02:32:33.109342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109321 4784 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 02:32:33.109431 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109422 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 02:32:33.109461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109436 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 02:32:33.109461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109442 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 02:32:33.109461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109448 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 02:32:33.109461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109453 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 02:32:33.109461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109459 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109464 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109470 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109477 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109482 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109496 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109506 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109529 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 02:32:33.109592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.109536 4784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 02:32:33.113125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.113106 4784 server.go:1263] "Started kubelet" Nov 04 02:32:33.113283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.113204 4784 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 02:32:33.113343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.113232 4784 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 02:32:33.113343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.113329 4784 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 02:32:33.113673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.113650 4784 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 02:32:33.113720 ip-10-0-133-209 systemd[1]: Started Kubernetes Kubelet. Nov 04 02:32:33.114658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.114643 4784 server.go:317] "Adding debug handlers to kubelet server" Nov 04 02:32:33.116880 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.116850 4784 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 02:32:33.116973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.116896 4784 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 02:32:33.116973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.116932 4784 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 02:32:33.116973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.116943 4784 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 02:32:33.117145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.117032 4784 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:52 +0000 UTC" deadline="2028-06-07 18:40:53.956829807 +0000 UTC" Nov 04 02:32:33.117145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.117065 4784 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13960h8m20.839767964s" Nov 04 02:32:33.117145 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.117100 4784 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.117941 4784 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120010 4784 factory.go:55] Registering systemd factory Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120036 4784 factory.go:223] Registration of the systemd container factory successfully Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120632 4784 factory.go:153] Registering CRI-O factory Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120647 4784 factory.go:223] Registration of the crio container factory successfully Nov 04 02:32:33.120710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120706 4784 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 02:32:33.121063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120726 4784 factory.go:103] Registering Raw factory Nov 04 02:32:33.121063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.120742 4784 manager.go:1199] Started watching for new ooms in manager Nov 04 02:32:33.121150 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.121085 4784 manager.go:319] Starting recovery of all containers Nov 04 02:32:33.124384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124335 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de579720-94f5-4cbd-b9f6-60e5b3d82647" volumeName="kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124392 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7e78e94-cc94-496f-ac33-f4e708709d9a" volumeName="kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124410 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124426 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124442 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f" volumeName="kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124457 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece428ad-e7c3-4744-925a-036290ed726e" volumeName="kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124471 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124491 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6d432e-6834-4230-9a45-57209c333b93" volumeName="kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124509 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124522 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7d9e-fc73-499b-9226-28e6b4c1d900" volumeName="kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh" seLinuxMountContext="" Nov 04 02:32:33.124539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124534 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7d9e-fc73-499b-9226-28e6b4c1d900" volumeName="kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124548 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" volumeName="kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124560 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2376a8d-13b4-490b-9f66-b55cf5f3a51f" volumeName="kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124592 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124608 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124621 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ccac4d-0b67-439b-a462-e11f04cdb14c" volumeName="kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124651 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78af83a8-adea-4abe-962e-c95203bdcde2" volumeName="kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124664 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0ca244-6231-4058-9051-8aa91ae0ce73" volumeName="kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124678 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0abdefaf-ed85-4c17-bfe7-11b0aec18b44" volumeName="kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124691 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9a6a6d-33fc-45ea-9b03-af9724367475" volumeName="kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124705 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="864630d2-d25b-4d12-aa4a-c9a60f5a13f4" volumeName="kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124717 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" volumeName="kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124729 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a19051-1ac9-4caf-82df-5a004c49db68" volumeName="kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124747 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584afc7e-72fd-4893-a77c-519e4b0b44ad" volumeName="kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124761 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7e78e94-cc94-496f-ac33-f4e708709d9a" volumeName="kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124773 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7e78e94-cc94-496f-ac33-f4e708709d9a" volumeName="kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124786 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06b3575-3fcd-4564-89b5-c7822dc36b59" volumeName="kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124833 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="399e71a6-e15f-4aea-b7fc-25f45d6c182e" volumeName="kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124854 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="399e71a6-e15f-4aea-b7fc-25f45d6c182e" volumeName="kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124868 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124903 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="864630d2-d25b-4d12-aa4a-c9a60f5a13f4" volumeName="kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124919 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124943 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea891a3-ac68-4f11-add3-b42a1b657846" volumeName="kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124956 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124969 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="399e71a6-e15f-4aea-b7fc-25f45d6c182e" volumeName="kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124980 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.124991 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0009777c-b015-471c-8d88-72d341de5778" volumeName="kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125009 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125021 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bb9090-5872-40b1-ae31-b910dc6575fa" volumeName="kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125032 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f" volumeName="kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125045 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="399e71a6-e15f-4aea-b7fc-25f45d6c182e" volumeName="kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 02:32:33.125041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125058 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125072 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2376a8d-13b4-490b-9f66-b55cf5f3a51f" volumeName="kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125089 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a19051-1ac9-4caf-82df-5a004c49db68" volumeName="kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125101 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de579720-94f5-4cbd-b9f6-60e5b3d82647" volumeName="kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125115 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece428ad-e7c3-4744-925a-036290ed726e" volumeName="kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125127 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95e3e816-aa1d-4022-b606-0b4ecc1ee79e" volumeName="kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125138 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bb9090-5872-40b1-ae31-b910dc6575fa" volumeName="kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125151 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de579720-94f5-4cbd-b9f6-60e5b3d82647" volumeName="kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125165 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125180 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bb9090-5872-40b1-ae31-b910dc6575fa" volumeName="kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125287 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a19051-1ac9-4caf-82df-5a004c49db68" volumeName="kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125300 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece428ad-e7c3-4744-925a-036290ed726e" volumeName="kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125314 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125327 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125340 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0abdefaf-ed85-4c17-bfe7-11b0aec18b44" volumeName="kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125361 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3967e0a1-3d8a-4036-b369-34302fb7ee32" volumeName="kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125373 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78af83a8-adea-4abe-962e-c95203bdcde2" volumeName="kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125386 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed" volumeName="kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125397 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06b3575-3fcd-4564-89b5-c7822dc36b59" volumeName="kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125408 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06b3575-3fcd-4564-89b5-c7822dc36b59" volumeName="kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125421 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0ca244-6231-4058-9051-8aa91ae0ce73" volumeName="kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125434 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de579720-94f5-4cbd-b9f6-60e5b3d82647" volumeName="kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125460 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" volumeName="kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125473 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0009777c-b015-471c-8d88-72d341de5778" volumeName="kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125485 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="220ee58a-5ca9-4f09-a317-e5b66524efb0" volumeName="kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125497 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="399e71a6-e15f-4aea-b7fc-25f45d6c182e" volumeName="kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125512 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a5e4c2-b127-4bfe-9a5b-943e48410287" volumeName="kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125525 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125538 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3264410-13ad-450b-9c9a-3d2583359b09" volumeName="kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125551 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584afc7e-72fd-4893-a77c-519e4b0b44ad" volumeName="kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125564 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ee7cb7-c423-4701-b7ab-3ee603748499" volumeName="kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125577 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0abdefaf-ed85-4c17-bfe7-11b0aec18b44" volumeName="kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125590 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d03902-4d33-4bec-9725-53ec99ca7dbd" volumeName="kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125602 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125615 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8071b266-034b-4622-822b-f8b01b65a685" volumeName="kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125626 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" volumeName="kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125639 4784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0ca244-6231-4058-9051-8aa91ae0ce73" volumeName="kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv" seLinuxMountContext="" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125651 4784 reconstruct.go:97] "Volume reconstruction finished" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.125659 4784 reconciler.go:29] "Reconciler: start to sync state" Nov 04 02:32:33.126480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.126130 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:33.127692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.127394 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:33.131021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.131000 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:33.132440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.132413 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:33.150000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.149955 4784 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 02:32:33.152267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.152242 4784 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 02:32:33.152267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.152275 4784 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 02:32:33.152461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.152297 4784 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 02:32:33.152461 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.152336 4784 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 02:32:33.174527 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.174498 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:33.174677 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.174536 4784 generic.go:356] "Generic (PLEG): container finished" podID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" exitCode=255 Nov 04 02:32:33.181562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.181537 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:32:33.181956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.181926 4784 generic.go:356] "Generic (PLEG): container finished" podID="399e71a6-e15f-4aea-b7fc-25f45d6c182e" containerID="bff232e2875d6d3346bea1674f1376e6baff81a3c20a757332e8b96c014a0b51" exitCode=1 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193504 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="e0889175e57a076873f6d7303ef3fc57c310579a0dad98390414ec2d1c5e1236" exitCode=0 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193536 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="db788d61a7d23e02451ac5eadc0862e6ca70ff6165e8e028e466dbdab3127cc5" exitCode=0 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193549 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="6a09ac27ff7569ae5beb339d46ad0b6a5b6ecd143d007ac3d06e29a71954603c" exitCode=0 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193557 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="1f1cc8091632e0ac27fe25325632b6e980091b712f3a67cdbf2dca7dae27b63c" exitCode=0 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193565 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="f0674b0ba8a9fb4c6c32dffda75bb6967f1cc894d88e414d33901f7f8f5452e5" exitCode=0 Nov 04 02:32:33.193579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.193573 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5d950c5-819d-4cc5-aa08-7f80bf099eb1" containerID="f8a443a540fdb2d7db5713b36538e66bb0e5d307196e85fdd75caa4a74e81008" exitCode=0 Nov 04 02:32:33.196921 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.196890 4784 generic.go:356] "Generic (PLEG): container finished" podID="0663bb7aee2243f0f782ca4003b09473" containerID="da789e0bc777e5dae368397004b4c5074f8318cacfa5909e51de27cd47f304e2" exitCode=0 Nov 04 02:32:33.208258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.208237 4784 manager.go:324] Recovery completed Nov 04 02:32:33.217653 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.217626 4784 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:32:33.225420 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.225398 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.226644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.226627 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.226734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.226662 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.226734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.226676 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.228627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228608 4784 cpu_manager.go:226] "Starting" policy="none" Nov 04 02:32:33.228627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228623 4784 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 02:32:33.228748 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228639 4784 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 02:32:33.228748 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228624 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.228808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228761 4784 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 02:32:33.228808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228771 4784 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 02:32:33.228808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228791 4784 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 02:32:33.228808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228797 4784 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 02:32:33.228808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228802 4784 policy_none.go:50] "Start" Nov 04 02:32:33.228936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228810 4784 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 02:32:33.228936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.228819 4784 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 02:32:33.229634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.229618 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.229681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.229651 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.229681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.229667 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.241893 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.241872 4784 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 02:32:33.241893 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.241893 4784 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 02:32:33.242005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.241901 4784 policy_none.go:44] "Start" Nov 04 02:32:33.246366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.246329 4784 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 02:32:33.248880 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.248864 4784 manager.go:341] "Starting Device Plugin manager" Nov 04 02:32:33.248974 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.248896 4784 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 02:32:33.248974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.248908 4784 server.go:87] "Starting device plugin registration server" Nov 04 02:32:33.249232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249219 4784 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 02:32:33.249308 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249234 4784 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 02:32:33.249354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249339 4784 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 02:32:33.249510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249462 4784 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 02:32:33.249510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249507 4784 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 02:32:33.249510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249514 4784 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 02:32:33.249683 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.249569 4784 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T02:32:33.249496592Z","UUID":"6e248c58-303f-4ae3-8686-b828465d726e","Handler":null,"Name":"","Endpoint":""} Nov 04 02:32:33.250866 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.250844 4784 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 02:32:33.250982 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.250884 4784 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" Nov 04 02:32:33.251049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.250980 4784 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 02:32:33.251049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.251007 4784 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 02:32:33.252500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.252462 4784 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal"] Nov 04 02:32:33.252630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.252608 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.254034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.254018 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.254112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.254041 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.254112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.254052 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.254463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.254448 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.254653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.254637 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.255256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255237 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.255339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255273 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.255339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255288 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.255339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255243 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.255339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255337 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.255506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255359 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.255542 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.255519 4784 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.255677 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255659 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.255776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255664 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerStarted","Data":"5a324a9ec436b3dafc983669fb6e4e233a9fbe2a8b687fce1093359ff621c4e6"} Nov 04 02:32:33.255776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255701 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerDied","Data":"da789e0bc777e5dae368397004b4c5074f8318cacfa5909e51de27cd47f304e2"} Nov 04 02:32:33.255776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255714 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"0663bb7aee2243f0f782ca4003b09473","Type":"ContainerStarted","Data":"41656babe496e599d32cc5da497eb55453ac0f97127c486ee343a41e7b18825f"} Nov 04 02:32:33.255776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255726 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"41843a288696f78001f75198b2c4caf0","Type":"ContainerStarted","Data":"36077d532873c0f9344963fd8a1ab8f8bc6e54c61d5c3cc21248058f3f31abbf"} Nov 04 02:32:33.255776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.255736 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" event={"ID":"41843a288696f78001f75198b2c4caf0","Type":"ContainerStarted","Data":"4b8dd54ac591a0da675c23bc3df2eb9a222201818b81699b1ac87ed60f254a41"} Nov 04 02:32:33.256449 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.256430 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.256517 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.256460 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.256517 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.256474 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.256622 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.256601 4784 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.us-west-1.compute.internal\" not found" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.317302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.317269 4784 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.317469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.317307 4784 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.317469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.317317 4784 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.317469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.317412 4784 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 02:32:33.319238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.319140 4784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.349435 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.349398 4784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.350601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.350580 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.350707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.350617 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.350707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.350631 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.350707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.350674 4784 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.368172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.368142 4784 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.381298 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.381272 4784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:33.381436 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.381319 4784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.381436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.381333 4784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.417555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.417523 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.417555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.417556 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.417710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.417578 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.470683 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.470611 4784 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.470802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.470723 4784 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.471365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.471339 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeNotReady" Nov 04 02:32:33.471417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.471379 4784 setters.go:546] "Node became not ready" node="ip-10-0-133-209.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T02:32:33Z","lastTransitionTime":"2026-11-04T02:32:33Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 02:32:33.483321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.483291 4784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:33.483434 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:33.483352 4784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.483666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.483651 4784 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-209.us-west-1.compute.internal" is owned by different node (oldNodeID="51b9baed-a92b-4b6f-9c45-82ee580997e8", newNodeID=""), cleaning up... Nov 04 02:32:33.517883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.517849 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.517883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.517870 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.517969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.517901 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.517969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.517934 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.517969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.517968 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41843a288696f78001f75198b2c4caf0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"41843a288696f78001f75198b2c4caf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.518112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.518032 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0663bb7aee2243f0f782ca4003b09473-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" (UID: \"0663bb7aee2243f0f782ca4003b09473\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:33.746277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.746173 4784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-209.us-west-1.compute.internal" event="NodeReady" Nov 04 02:32:33.746407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:33.746287 4784 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 02:32:34.096852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.096755 4784 apiserver.go:52] "Watching apiserver" Nov 04 02:32:34.204747 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.204714 4784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:34.204912 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.204893 4784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:34.254427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.254396 4784 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 02:32:34.255323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.255300 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc","openshift-ingress/router-default-76598c4567-mhntk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn","kube-system/global-pull-secret-syncer-65l2h","openshift-ingress-canary/ingress-canary-89rsm","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6","kube-system/kas-connection-checker-685895c7c9-8cz2p","openshift-insights/insights-operator-7c4986d449-nxqpl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2","openshift-multus/multus-k6jrq","kube-system/konnectivity-agent-qm2lj","openshift-network-operator/iptables-alerter-mcv6p","openshift-network-console/networking-console-plugin-86c478bb88-l2nk6","openshift-dns/dns-default-h9cmn","kube-system/kas-connection-checker-685895c7c9-bzwxn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g","openshift-cluster-node-tuning-operator/tuned-wrn76","openshift-insights/insights-runtime-extractor-hl5s9","openshift-ovn-kubernetes/ovnkube-node-82r7r","openshift-console-operator/console-operator-7cd56b8dbc-zp9vk","openshift-image-registry/image-registry-7b7c9cfd78-tdjrx","openshift-network-diagnostics/network-check-target-4xfw5","openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9","openshift-image-registry/node-ca-txm72","openshift-multus/multus-additional-cni-plugins-rsmrc","openshift-multus/network-metrics-daemon-mbppk","openshift-image-registry/image-registry-7947ff7c95-mx6lt","kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal","openshift-dns/node-resolver-8m8pf","openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2","kube-system/kas-connection-checker-685895c7c9-8xkb7"] Nov 04 02:32:34.256792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.256772 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.257991 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.257966 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.258095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.258051 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.258803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.258785 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.262333 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.262308 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.263281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.263260 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.267523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.267501 4784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:34.268084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.268068 4784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:34.268336 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.268324 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.269554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.269534 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.270807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.270789 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.304709 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.304662 4784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:34.304709 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.304677 4784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal" Nov 04 02:32:34.312159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312125 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 02:32:34.312159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312124 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 02:32:34.312159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312138 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 02:32:34.312544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312225 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 02:32:34.312544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312418 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 02:32:34.312544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312440 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 02:32:34.312544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312475 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drzkl\"" Nov 04 02:32:34.312697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312579 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xmfmq\"" Nov 04 02:32:34.312736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312722 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 02:32:34.312822 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312808 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 02:32:34.312870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.312858 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 02:32:34.313381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.313371 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 02:32:34.313508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.313498 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 02:32:34.315306 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315292 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.315698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315678 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.315849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315718 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.315849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315773 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.315849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315794 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315884 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315926 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315985 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.315992 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316005 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316010 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316024 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316028 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316025 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316060 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316064 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316073 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316081 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316102 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316074 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316173 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316213 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316256 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316410 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.316402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316457 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.317100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316459 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 02:32:34.317100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316494 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 02:32:34.317100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316539 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 02:32:34.317100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316667 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.317100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.316738 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mk8j7\"" Nov 04 02:32:34.320434 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.320416 4784 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 02:32:34.321730 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.321713 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.321780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.321740 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.321780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.321760 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.321842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.321785 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.322230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322215 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.322269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322246 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsvc\" (UniqueName: \"kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.322326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322308 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:34.322397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322378 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.322432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322391 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de579720-94f5-4cbd-b9f6-60e5b3d82647-serving-cert\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.322432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322410 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.322432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322421 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0009777c-b015-471c-8d88-72d341de5778-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.322534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322447 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.322534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322477 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.322534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322486 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a19051-1ac9-4caf-82df-5a004c49db68-data-volume\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.322534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322500 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.322534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322523 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322542 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322589 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322623 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322653 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322683 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322710 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322715 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-tuned\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.322757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322735 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322773 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322803 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322891 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78af83a8-adea-4abe-962e-c95203bdcde2-tmp-dir\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322914 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttlh\" (UniqueName: \"kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322949 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322977 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9k6\" (UniqueName: \"kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6\") pod \"network-check-source-6f78d59f99-hm2jc\" (UID: \"3967e0a1-3d8a-4036-b369-34302fb7ee32\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.322996 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1bb9090-5872-40b1-ae31-b910dc6575fa-config\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323020 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.323067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323052 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323077 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323080 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323110 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323155 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323180 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323240 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323309 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323387 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.323417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323417 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323487 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323487 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323540 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r2t\" (UniqueName: \"kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323567 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323594 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323626 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323650 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323677 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323687 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323703 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323736 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.323788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323765 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323797 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323812 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323833 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtptf\" (UniqueName: \"kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323849 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323891 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tmv\" (UniqueName: \"kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323906 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323940 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.323989 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwq9\" (UniqueName: \"kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.324100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324086 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324106 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566wr\" (UniqueName: \"kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324149 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324155 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06b3575-3fcd-4564-89b5-c7822dc36b59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324202 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324232 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324255 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwfc\" (UniqueName: \"kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc\") pod \"volume-data-source-validator-58c9f5f46d-2chk6\" (UID: \"e5ee7cb7-c423-4701-b7ab-3ee603748499\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324276 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324310 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.324363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324342 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfchl\" (UniqueName: \"kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl\") pod \"kas-connection-checker-685895c7c9-8cz2p\" (UID: \"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed\") " pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324375 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324399 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324427 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324534 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-service-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324534 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324567 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.324612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324593 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324679 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324712 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324729 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324774 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324800 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9plm\" (UniqueName: \"kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.324820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324819 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.325024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324869 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.325024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324921 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b3575-3fcd-4564-89b5-c7822dc36b59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.325024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324933 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pvs\" (UniqueName: \"kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.325024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.324977 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrspf\" (UniqueName: \"kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.325132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325073 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.325162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325150 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325227 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.325282 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325266 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnfj\" (UniqueName: \"kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.325323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325308 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxsj8\" (UniqueName: \"kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.325357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325330 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-default-certificate\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325371 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:34.325433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325418 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.325466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325457 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9vn\" (UniqueName: \"kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.325511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325466 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-snapshots\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.325511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325502 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.325609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325587 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325625 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.325703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.325703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325686 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.325782 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325704 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325782 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.325782 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325775 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325807 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325833 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwdf7\" (UniqueName: \"kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325865 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7qp\" (UniqueName: \"kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp\") pod \"kas-connection-checker-685895c7c9-8xkb7\" (UID: \"64d03902-4d33-4bec-9725-53ec99ca7dbd\") " pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325892 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325907 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-stats-auth\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.325929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325920 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325956 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325975 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.325992 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326032 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326072 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-tmp\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326080 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326107 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326123 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326143 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326168 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.326264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326244 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326277 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbjk\" (UniqueName: \"kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk\") pod \"kas-connection-checker-685895c7c9-bzwxn\" (UID: \"6e9a6a6d-33fc-45ea-9b03-af9724367475\") " pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326310 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326316 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-config\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326247 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326338 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326375 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptjw\" (UniqueName: \"kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326384 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326419 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326465 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326495 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326499 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326512 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326530 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326561 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326593 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254jx\" (UniqueName: \"kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326628 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326647 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326677 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326693 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.326705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326707 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff6d432e-6834-4230-9a45-57209c333b93-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326759 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea891a3-ac68-4f11-add3-b42a1b657846-tmp\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326802 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326839 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvskz\" (UniqueName: \"kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326858 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326920 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.326956 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1bb9090-5872-40b1-ae31-b910dc6575fa-serving-cert\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327017 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327042 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327060 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327096 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327135 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327200 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327241 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwhl\" (UniqueName: \"kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327268 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327294 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327389 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327409 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327430 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327451 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ea891a3-ac68-4f11-add3-b42a1b657846-serving-cert\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327455 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327497 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece428ad-e7c3-4744-925a-036290ed726e-tmp-dir\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327500 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327568 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-serviceca\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.327651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.327628 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.361451 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.361385 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 02:32:34.361451 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.361411 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.363660 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.363368 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.363813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.363698 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.366980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.364012 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.368441 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.368416 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 02:32:34.368950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.368933 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 02:32:34.368950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.368945 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 02:32:34.374745 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.374727 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ea891a3-ac68-4f11-add3-b42a1b657846-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.374813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.374737 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de579720-94f5-4cbd-b9f6-60e5b3d82647-trusted-ca\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.375913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.375895 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.376157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.376143 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.382752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.382734 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsvc\" (UniqueName: \"kubernetes.io/projected/de579720-94f5-4cbd-b9f6-60e5b3d82647-kube-api-access-7wsvc\") pod \"console-operator-7cd56b8dbc-zp9vk\" (UID: \"de579720-94f5-4cbd-b9f6-60e5b3d82647\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:34.382892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.382873 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9k6\" (UniqueName: \"kubernetes.io/projected/3967e0a1-3d8a-4036-b369-34302fb7ee32-kube-api-access-4n9k6\") pod \"network-check-source-6f78d59f99-hm2jc\" (UID: \"3967e0a1-3d8a-4036-b369-34302fb7ee32\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hm2jc" Nov 04 02:32:34.383546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.383531 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtptf\" (UniqueName: \"kubernetes.io/projected/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-api-access-rtptf\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.383794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.383771 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwq9\" (UniqueName: \"kubernetes.io/projected/a06b3575-3fcd-4564-89b5-c7822dc36b59-kube-api-access-8xwq9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2\" (UID: \"a06b3575-3fcd-4564-89b5-c7822dc36b59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" Nov 04 02:32:34.383966 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.383952 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r2t\" (UniqueName: \"kubernetes.io/projected/3ea891a3-ac68-4f11-add3-b42a1b657846-kube-api-access-l8r2t\") pod \"insights-operator-7c4986d449-nxqpl\" (UID: \"3ea891a3-ac68-4f11-add3-b42a1b657846\") " pod="openshift-insights/insights-operator-7c4986d449-nxqpl" Nov 04 02:32:34.384572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.384557 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwfc\" (UniqueName: \"kubernetes.io/projected/e5ee7cb7-c423-4701-b7ab-3ee603748499-kube-api-access-ldwfc\") pod \"volume-data-source-validator-58c9f5f46d-2chk6\" (UID: \"e5ee7cb7-c423-4701-b7ab-3ee603748499\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2chk6" Nov 04 02:32:34.387707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.387692 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.387754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.387727 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfchl\" (UniqueName: \"kubernetes.io/projected/8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed-kube-api-access-nfchl\") pod \"kas-connection-checker-685895c7c9-8cz2p\" (UID: \"8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed\") " pod="kube-system/kas-connection-checker-685895c7c9-8cz2p" Nov 04 02:32:34.396073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.396050 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7qp\" (UniqueName: \"kubernetes.io/projected/64d03902-4d33-4bec-9725-53ec99ca7dbd-kube-api-access-lx7qp\") pod \"kas-connection-checker-685895c7c9-8xkb7\" (UID: \"64d03902-4d33-4bec-9725-53ec99ca7dbd\") " pod="kube-system/kas-connection-checker-685895c7c9-8xkb7" Nov 04 02:32:34.396906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.396873 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9vn\" (UniqueName: \"kubernetes.io/projected/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-kube-api-access-sg9vn\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.396999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.396932 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254jx\" (UniqueName: \"kubernetes.io/projected/b1bb9090-5872-40b1-ae31-b910dc6575fa-kube-api-access-254jx\") pod \"service-ca-operator-544d749c8c-tbjj2\" (UID: \"b1bb9090-5872-40b1-ae31-b910dc6575fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397163 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397289 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397288 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbjk\" (UniqueName: \"kubernetes.io/projected/6e9a6a6d-33fc-45ea-9b03-af9724367475-kube-api-access-7xbjk\") pod \"kas-connection-checker-685895c7c9-bzwxn\" (UID: \"6e9a6a6d-33fc-45ea-9b03-af9724367475\") " pod="kube-system/kas-connection-checker-685895c7c9-bzwxn" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397439 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397463 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvskz\" (UniqueName: \"kubernetes.io/projected/0009777c-b015-471c-8d88-72d341de5778-kube-api-access-kvskz\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.397519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397508 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptjw\" (UniqueName: \"kubernetes.io/projected/f7e78e94-cc94-496f-ac33-f4e708709d9a-kube-api-access-4ptjw\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.397867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397557 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-kube-api-access-t2bvr\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.397867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.397807 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9f9\" (UniqueName: \"kubernetes.io/projected/59ccac4d-0b67-439b-a462-e11f04cdb14c-kube-api-access-8g9f9\") pod \"network-check-target-4xfw5\" (UID: \"59ccac4d-0b67-439b-a462-e11f04cdb14c\") " pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:34.411435 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.411409 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 02:32:34.411753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.411720 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 02:32:34.411870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.411850 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgzgm\"" Nov 04 02:32:34.411928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.411891 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2vt\"" Nov 04 02:32:34.411928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.411900 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjd4x\"" Nov 04 02:32:34.412362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412346 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6tfxk\"" Nov 04 02:32:34.412422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412370 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 02:32:34.412422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412385 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vd7gf\"" Nov 04 02:32:34.412422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412400 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 02:32:34.412422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412417 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412413 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm8qv\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412458 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-82bnw\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412462 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l8p2p\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412472 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6l7l7\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412472 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lg9c8\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412492 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412509 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412517 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4v2qx\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412525 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ffw8m\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412509 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmwv6\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412509 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9fmkz\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412543 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 02:32:34.412612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412592 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8zwfm\"" Nov 04 02:32:34.413006 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.412643 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-cert\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:34.413870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.413851 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-agent-certs\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:34.413905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.413870 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovn-node-metrics-cert\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.414157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414144 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414522 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414556 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414559 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414578 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414593 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.414621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414601 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414647 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414682 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414686 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414705 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414722 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414725 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414723 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414732 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 02:32:34.414875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414742 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.414961 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415002 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415028 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415047 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415053 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415083 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415094 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.415154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415088 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.415426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415226 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 02:32:34.415426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.415262 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/584afc7e-72fd-4893-a77c-519e4b0b44ad-iptables-alerter-script\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.416348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416333 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ece428ad-e7c3-4744-925a-036290ed726e-config-volume\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.416413 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416395 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a19051-1ac9-4caf-82df-5a004c49db68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.416447 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416406 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-script-lib\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.416590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416576 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2376a8d-13b4-490b-9f66-b55cf5f3a51f-konnectivity-ca\") pod \"konnectivity-agent-qm2lj\" (UID: \"b2376a8d-13b4-490b-9f66-b55cf5f3a51f\") " pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:32:34.416714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416701 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwdf7\" (UniqueName: \"kubernetes.io/projected/78af83a8-adea-4abe-962e-c95203bdcde2-kube-api-access-vwdf7\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.416749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.416737 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-original-pull-secret\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.417860 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.417837 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrspf\" (UniqueName: \"kubernetes.io/projected/82a5e4c2-b127-4bfe-9a5b-943e48410287-kube-api-access-wrspf\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.419985 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.419966 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxsj8\" (UniqueName: \"kubernetes.io/projected/220ee58a-5ca9-4f09-a317-e5b66524efb0-kube-api-access-dxsj8\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.419985 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.419974 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnfj\" (UniqueName: \"kubernetes.io/projected/ece428ad-e7c3-4744-925a-036290ed726e-kube-api-access-lgnfj\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.420081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.420011 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9plm\" (UniqueName: \"kubernetes.io/projected/399e71a6-e15f-4aea-b7fc-25f45d6c182e-kube-api-access-v9plm\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.422675 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.422657 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwhl\" (UniqueName: \"kubernetes.io/projected/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-kube-api-access-xqwhl\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.422840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.422825 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-cni-binary-copy\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.422885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.422827 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.423219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.423201 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttlh\" (UniqueName: \"kubernetes.io/projected/8cdf7d9e-fc73-499b-9226-28e6b4c1d900-kube-api-access-xttlh\") pod \"ingress-canary-89rsm\" (UID: \"8cdf7d9e-fc73-499b-9226-28e6b4c1d900\") " pod="openshift-ingress-canary/ingress-canary-89rsm" Nov 04 02:32:34.424153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424133 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.424153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424146 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-daemon-config\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.424295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424141 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-ovnkube-config\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.424295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424234 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/399e71a6-e15f-4aea-b7fc-25f45d6c182e-env-overrides\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.424295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424263 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.424560 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424544 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tmv\" (UniqueName: \"kubernetes.io/projected/cb0ca244-6231-4058-9051-8aa91ae0ce73-kube-api-access-g8tmv\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.424684 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424665 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pvs\" (UniqueName: \"kubernetes.io/projected/584afc7e-72fd-4893-a77c-519e4b0b44ad-kube-api-access-b2pvs\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.424744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.424727 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566wr\" (UniqueName: \"kubernetes.io/projected/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-kube-api-access-566wr\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.428830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428814 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.428887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.428887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428853 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.428887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428868 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.428887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428885 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428900 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428914 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428918 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-var-lib-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428926 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-var-lib-kubelet\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428929 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428941 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428954 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-slash\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428966 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428971 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-kubelet\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428968 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-kubernetes\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.428990 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429008 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429016 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429020 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429044 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.429056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429054 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-bin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429020 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysconfig\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429080 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429098 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929077253 +0000 UTC m=+1.895578245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429112 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-netns\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429124 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429144 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429159 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429174 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429205 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429213 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-ovn\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429220 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429223 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-system-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429246 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429252 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-log-socket\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429268 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-multus-certs\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429278 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-k8s-cni-cncf-io\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429288 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429270 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429302 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429309 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-sys\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429324 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429360 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929344754 +0000 UTC m=+1.895845742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429359 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-lib-modules\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429385 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-host\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429378 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429407 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429429 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429460 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429483 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429506 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429527 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-run\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429530 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429509 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-cnibin\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429539 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-run-netns\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429554 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429572 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429581 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-kubelet\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429582 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429610 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429626 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429626 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-device-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429641 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429649 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-sysctl-conf\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429660 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429679 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a19051-1ac9-4caf-82df-5a004c49db68-crio-socket\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429684 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429686 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-run-ovn-kubernetes\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429701 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429705 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429716 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429733 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429746 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-modprobe-d\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429735 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429763 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929746155 +0000 UTC m=+1.896247149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429783 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-etc-openvswitch\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.429752 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429790 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929779038 +0000 UTC m=+1.896280009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429809 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429827 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429793 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-netd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429848 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429812 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429870 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-cni-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429885 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429891 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929881319 +0000 UTC m=+1.896382290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429901 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-socket-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.429913 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.929904132 +0000 UTC m=+1.896405145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429927 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78af83a8-adea-4abe-962e-c95203bdcde2-hosts-file\") pod \"node-resolver-8m8pf\" (UID: \"78af83a8-adea-4abe-962e-c95203bdcde2\") " pod="openshift-dns/node-resolver-8m8pf" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429929 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429962 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429968 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-registration-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.429986 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430003 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/864630d2-d25b-4d12-aa4a-c9a60f5a13f4-host\") pod \"node-ca-txm72\" (UID: \"864630d2-d25b-4d12-aa4a-c9a60f5a13f4\") " pod="openshift-image-registry/node-ca-txm72" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430009 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430025 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430041 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430077 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584afc7e-72fd-4893-a77c-519e4b0b44ad-host-slash\") pod \"iptables-alerter-mcv6p\" (UID: \"584afc7e-72fd-4893-a77c-519e4b0b44ad\") " pod="openshift-network-operator/iptables-alerter-mcv6p" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430081 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-dbus\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430086 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430088 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-hostroot\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430110 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430126 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430145 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430157 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430153 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430201 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d950c5-819d-4cc5-aa08-7f80bf099eb1-cnibin\") pod \"multus-additional-cni-plugins-rsmrc\" (UID: \"a5d950c5-819d-4cc5-aa08-7f80bf099eb1\") " pod="openshift-multus/multus-additional-cni-plugins-rsmrc" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430217 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.93020481 +0000 UTC m=+1.896705813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430201 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-node-log\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430252 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430267 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430281 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430288 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdefaf-ed85-4c17-bfe7-11b0aec18b44-etc-systemd\") pod \"tuned-wrn76\" (UID: \"0abdefaf-ed85-4c17-bfe7-11b0aec18b44\") " pod="openshift-cluster-node-tuning-operator/tuned-wrn76" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430295 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430309 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430323 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430339 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-run-systemd\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430341 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430370 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-host-cni-bin\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430375 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-host-var-lib-cni-multus\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430385 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430392 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e3e816-aa1d-4022-b606-0b4ecc1ee79e-kubelet-config\") pod \"global-pull-secret-syncer-65l2h\" (UID: \"95e3e816-aa1d-4022-b606-0b4ecc1ee79e\") " pod="kube-system/global-pull-secret-syncer-65l2h" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430401 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430407 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-socket-dir-parent\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430404 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430438 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.930425755 +0000 UTC m=+1.896926739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430436 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430449 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430455 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430471 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.930458233 +0000 UTC m=+1.896959203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430485 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.430489 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:34.930478911 +0000 UTC m=+1.896979884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430510 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430534 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430545 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-multus-conf-dir\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430553 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430572 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/399e71a6-e15f-4aea-b7fc-25f45d6c182e-systemd-units\") pod \"ovnkube-node-82r7r\" (UID: \"399e71a6-e15f-4aea-b7fc-25f45d6c182e\") " pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430581 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430613 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a5e4c2-b127-4bfe-9a5b-943e48410287-sys-fs\") pod \"aws-ebs-csi-driver-node-pxl4g\" (UID: \"82a5e4c2-b127-4bfe-9a5b-943e48410287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxl4g" Nov 04 02:32:34.431591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.430646 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb0ca244-6231-4058-9051-8aa91ae0ce73-etc-kubernetes\") pod \"multus-k6jrq\" (UID: \"cb0ca244-6231-4058-9051-8aa91ae0ce73\") " pod="openshift-multus/multus-k6jrq" Nov 04 02:32:34.558506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.558376 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:32:34.560613 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.560594 4784 scope.go:122] "RemoveContainer" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" Nov 04 02:32:34.935668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935634 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935681 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935713 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935733 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935760 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935783 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:34.935832 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935803 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935857 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935872 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935876 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935859 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935867 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.935852014 +0000 UTC m=+2.902352988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935939 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.935984 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.935998 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936019 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.936004 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936026 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936011023 +0000 UTC m=+2.902512002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936049 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936059 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936045185 +0000 UTC m=+2.902546163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936061 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936078 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.93606775 +0000 UTC m=+2.902568719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936099 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936088253 +0000 UTC m=+2.902589222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936115 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936109655 +0000 UTC m=+2.902610625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:34.936113 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936126 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.93612154 +0000 UTC m=+2.902622510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936138 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936132974 +0000 UTC m=+2.902633943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.936158 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:34.936181 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936284 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936300 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936289393 +0000 UTC m=+2.902790369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:34.936739 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:34.936329 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.936312625 +0000 UTC m=+2.902813617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:35.047653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.047616 4784 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:35.209251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.209164 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xfw5" event={"ID":"59ccac4d-0b67-439b-a462-e11f04cdb14c","Type":"ContainerStarted","Data":"fba1ae3313049023573303665852448e740ed32ecb498b3f743494369741cb8d"} Nov 04 02:32:35.210897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.210874 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/2.log" Nov 04 02:32:35.211339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.211321 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/1.log" Nov 04 02:32:35.211409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.211366 4784 generic.go:356] "Generic (PLEG): container finished" podID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" exitCode=255 Nov 04 02:32:35.211458 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.211442 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerDied","Data":"fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7"} Nov 04 02:32:35.211515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.211501 4784 scope.go:122] "RemoveContainer" containerID="d22cd2c55306000423b3dcb90ebad409119c8e01fbbb75f8e68ac8eb8f184c51" Nov 04 02:32:35.211732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.211715 4784 scope.go:122] "RemoveContainer" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" Nov 04 02:32:35.211957 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.211938 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:35.946316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946257 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946338 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946373 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946387 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946400 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946456 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946424117 +0000 UTC m=+4.912925120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946476 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946483 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946486 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946489 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946502 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:35.946518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946510 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946529 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.94651462 +0000 UTC m=+4.913015614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946550 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.94654138 +0000 UTC m=+4.913042355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946565 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946560187 +0000 UTC m=+4.913061156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946586 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946591 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946581479 +0000 UTC m=+4.913082482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946631 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946621677 +0000 UTC m=+4.913122647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946655 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946679 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946699 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:35.946731 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946790 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946800 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946808 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946812 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946814 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946831 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.94682091 +0000 UTC m=+4.913321883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946850 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946838521 +0000 UTC m=+4.913339541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:35.946875 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946871 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946856756 +0000 UTC m=+4.913357727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:35.947425 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:35.946898 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:37.946883154 +0000 UTC m=+4.913384130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:36.222598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:36.222519 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/2.log" Nov 04 02:32:36.223043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:36.222839 4784 scope.go:122] "RemoveContainer" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" Nov 04 02:32:36.223043 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:36.223016 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:37.966353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966315 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:37.966353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966364 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966395 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966426 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966458 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966479 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966489 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966487 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966509 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966528 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966558 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966537558 +0000 UTC m=+8.933038541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966557 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966586 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966593 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966575673 +0000 UTC m=+8.933076645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966595 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966560 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966604 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966617 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.96660713 +0000 UTC m=+8.933108102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966642 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966628467 +0000 UTC m=+8.933129439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966657 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966651088 +0000 UTC m=+8.933152057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966670 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966664726 +0000 UTC m=+8.933165696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966684 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966676393 +0000 UTC m=+8.933177363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966712 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966733 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:37.966751 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966804 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:37.966809 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966814 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:37.967561 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966833 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:37.967561 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966841 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966830585 +0000 UTC m=+8.933331556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:37.967561 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966860 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966850865 +0000 UTC m=+8.933351869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:37.967561 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:37.966873 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:41.966867422 +0000 UTC m=+8.933368391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:38.621056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:38.621014 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:38.621460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:38.621446 4784 scope.go:122] "RemoveContainer" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" Nov 04 02:32:38.621654 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:38.621634 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:39.246503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:39.246474 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:39.979543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:39.979498 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:32:42.009161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009108 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:42.009161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009173 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009233 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009260 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009291 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009314 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009361 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009380 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009386 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009396 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009375644 +0000 UTC m=+16.975876636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009411 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009417 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009404303 +0000 UTC m=+16.975905273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009322 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009436 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009431043 +0000 UTC m=+16.975932013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009415 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009465 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009449817 +0000 UTC m=+16.975950791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009480 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009474086 +0000 UTC m=+16.975975056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009493 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009486847 +0000 UTC m=+16.975987817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009530 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009556 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009578 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:42.009597 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009645 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009658 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009661 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009677 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009688 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.00967876 +0000 UTC m=+16.976179766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009665 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:42.009678 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009705 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009698956 +0000 UTC m=+16.976199927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:42.010491 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009715 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009709702 +0000 UTC m=+16.976210729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:42.010491 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:42.009726 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.009720152 +0000 UTC m=+16.976221122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:43.366705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:43.366675 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:32:43.367181 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:43.366891 4784 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 02:32:45.047877 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.047839 4784 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:45.048286 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.048254 4784 scope.go:122] "RemoveContainer" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" Nov 04 02:32:45.252586 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.252557 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:32:45.253088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.253071 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/2.log" Nov 04 02:32:45.253170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.253106 4784 generic.go:356] "Generic (PLEG): container finished" podID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerID="a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b" exitCode=255 Nov 04 02:32:45.253243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.253201 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerDied","Data":"a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b"} Nov 04 02:32:45.253292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.253270 4784 scope.go:122] "RemoveContainer" containerID="fc8db3031335448fdb1b33ab12ca305c4494de8c7cbb4ceaa1c8ac7baece20a7" Nov 04 02:32:45.253629 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.253609 4784 scope.go:122] "RemoveContainer" containerID="a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b" Nov 04 02:32:45.253823 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:45.253804 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:45.255539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:45.255514 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" event={"ID":"a06b3575-3fcd-4564-89b5-c7822dc36b59","Type":"ContainerStarted","Data":"058fb1ea720add341b4db63342d1649f0f67a47fa6459d20a2852b067c666321"} Nov 04 02:32:46.070002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.069915 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k"] Nov 04 02:32:46.071862 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.071845 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" Nov 04 02:32:46.129499 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.129469 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k"] Nov 04 02:32:46.153240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.153206 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xnw\" (UniqueName: \"kubernetes.io/projected/bb76e13a-ad66-45d7-9ad1-812c1cf67042-kube-api-access-c2xnw\") pod \"migrator-66df5d496b-vxj7k\" (UID: \"bb76e13a-ad66-45d7-9ad1-812c1cf67042\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" Nov 04 02:32:46.215945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.215908 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:46.215945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.215916 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p5kw\"" Nov 04 02:32:46.216341 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.215954 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 02:32:46.253949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.253910 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xnw\" (UniqueName: \"kubernetes.io/projected/bb76e13a-ad66-45d7-9ad1-812c1cf67042-kube-api-access-c2xnw\") pod \"migrator-66df5d496b-vxj7k\" (UID: \"bb76e13a-ad66-45d7-9ad1-812c1cf67042\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" Nov 04 02:32:46.259733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.259710 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:32:46.261204 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.261160 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" event={"ID":"b1bb9090-5872-40b1-ae31-b910dc6575fa","Type":"ContainerStarted","Data":"7de19416eb5e6e4670dee89edf0be7bf150ec36fa5b2e3fe6ff5e7ec847293e9"} Nov 04 02:32:46.313717 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.313686 4784 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 02:32:46.316168 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.316138 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xnw\" (UniqueName: \"kubernetes.io/projected/bb76e13a-ad66-45d7-9ad1-812c1cf67042-kube-api-access-c2xnw\") pod \"migrator-66df5d496b-vxj7k\" (UID: \"bb76e13a-ad66-45d7-9ad1-812c1cf67042\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" Nov 04 02:32:46.381915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.381817 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" Nov 04 02:32:46.759719 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:46.759695 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k"] Nov 04 02:32:46.762019 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:32:46.761992 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb76e13a_ad66_45d7_9ad1_812c1cf67042.slice/crio-7eaf2196a6d9e448e6cb95bd3ac834ceb655fa428ffa1d4f2d1289cfaffafbb6 WatchSource:0}: Error finding container 7eaf2196a6d9e448e6cb95bd3ac834ceb655fa428ffa1d4f2d1289cfaffafbb6: Status 404 returned error can't find the container with id 7eaf2196a6d9e448e6cb95bd3ac834ceb655fa428ffa1d4f2d1289cfaffafbb6 Nov 04 02:32:47.265744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:47.265709 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" event={"ID":"bb76e13a-ad66-45d7-9ad1-812c1cf67042","Type":"ContainerStarted","Data":"7eaf2196a6d9e448e6cb95bd3ac834ceb655fa428ffa1d4f2d1289cfaffafbb6"} Nov 04 02:32:47.566902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:47.566826 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8m8pf_78af83a8-adea-4abe-962e-c95203bdcde2/dns-node-resolver/0.log" Nov 04 02:32:48.622091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:48.622049 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:48.622556 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:48.622536 4784 scope.go:122] "RemoveContainer" containerID="a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b" Nov 04 02:32:48.622807 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:48.622781 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:32:48.876436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:48.876358 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-txm72_864630d2-d25b-4d12-aa4a-c9a60f5a13f4/node-ca/0.log" Nov 04 02:32:49.273336 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:49.273297 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" event={"ID":"bb76e13a-ad66-45d7-9ad1-812c1cf67042","Type":"ContainerStarted","Data":"a76ae38545d3335667c9062de2d24d6c6bf2e31188f2370dca65ff28ab630af2"} Nov 04 02:32:49.276162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:49.276137 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-89rsm_8cdf7d9e-fc73-499b-9226-28e6b4c1d900/serve-healthcheck-canary/0.log" Nov 04 02:32:49.394302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:49.394247 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vxj7k" podStartSLOduration=1.733203609 podStartE2EDuration="3.394232403s" totalImagesPullingTime="1.661028794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:49.393650346 +0000 UTC m=+16.360151338" watchObservedRunningTime="2026-11-04 02:32:49.394232403 +0000 UTC m=+16.360733391" Nov 04 02:32:50.090958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.090920 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:50.090958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.090969 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.090993 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091065 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091100 4784 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091103 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091135 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091109299 +0000 UTC m=+33.057610285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : configmap references non-existent config key: service-ca.crt Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091150 4784 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091158 4784 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091172 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls podName:ece428ad-e7c3-4744-925a-036290ed726e nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091161923 +0000 UTC m=+33.057662893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls") pod "dns-default-h9cmn" (UID: "ece428ad-e7c3-4744-925a-036290ed726e") : secret "dns-default-metrics-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091174 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091213 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7c9cfd78-tdjrx: secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091225 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls podName:0009777c-b015-471c-8d88-72d341de5778 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091208446 +0000 UTC m=+33.057709460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9lmn" (UID: "0009777c-b015-471c-8d88-72d341de5778") : secret "samples-operator-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091254 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert podName:ff6d432e-6834-4230-9a45-57209c333b93 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091245433 +0000 UTC m=+33.057746408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l2nk6" (UID: "ff6d432e-6834-4230-9a45-57209c333b93") : secret "networking-console-plugin-cert" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091265 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls podName:8071b266-034b-4622-822b-f8b01b65a685 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091260711 +0000 UTC m=+33.057761681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls") pod "image-registry-7b7c9cfd78-tdjrx" (UID: "8071b266-034b-4622-822b-f8b01b65a685") : secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091302 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091334 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091372 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091394 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:50.091419 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091445 4784 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091454 4784 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091458 4784 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091468 4784 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7947ff7c95-mx6lt: secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091488 4784 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091488 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls podName:c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.09147517 +0000 UTC m=+33.057976139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tjjj9" (UID: "c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f") : secret "cluster-monitoring-operator-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091512 4784 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091531 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs podName:220ee58a-5ca9-4f09-a317-e5b66524efb0 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091524198 +0000 UTC m=+33.058025167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs") pod "network-metrics-daemon-mbppk" (UID: "220ee58a-5ca9-4f09-a317-e5b66524efb0") : secret "metrics-daemon-secret" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091541 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls podName:f3264410-13ad-450b-9c9a-3d2583359b09 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091536199 +0000 UTC m=+33.058037169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls") pod "image-registry-7947ff7c95-mx6lt" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09") : secret "image-registry-tls" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091553 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs podName:f7e78e94-cc94-496f-ac33-f4e708709d9a nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.091547302 +0000 UTC m=+33.058048272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs") pod "router-default-76598c4567-mhntk" (UID: "f7e78e94-cc94-496f-ac33-f4e708709d9a") : secret "router-metrics-certs-default" not found Nov 04 02:32:50.091554 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:50.091578 4784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls podName:c1a19051-1ac9-4caf-82df-5a004c49db68 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.09155928 +0000 UTC m=+33.058060254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl5s9" (UID: "c1a19051-1ac9-4caf-82df-5a004c49db68") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:55.047909 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:55.047866 4784 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:32:55.048332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:32:55.048282 4784 scope.go:122] "RemoveContainer" containerID="a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b" Nov 04 02:32:55.048476 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:32:55.048454 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zp9vk_openshift-console-operator(de579720-94f5-4cbd-b9f6-60e5b3d82647)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" Nov 04 02:33:00.464465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:00.464437 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82r7r" Nov 04 02:33:02.424910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:02.424880 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qm2lj" Nov 04 02:33:06.141515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141473 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141533 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141565 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141598 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141633 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141678 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141716 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141747 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141774 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:33:06.142017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.141826 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:33:06.142705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.142490 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e78e94-cc94-496f-ac33-f4e708709d9a-service-ca-bundle\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:06.144993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.144856 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220ee58a-5ca9-4f09-a317-e5b66524efb0-metrics-certs\") pod \"network-metrics-daemon-mbppk\" (UID: \"220ee58a-5ca9-4f09-a317-e5b66524efb0\") " pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:33:06.144993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.144949 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"image-registry-7b7c9cfd78-tdjrx\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:06.144993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.144962 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a19051-1ac9-4caf-82df-5a004c49db68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl5s9\" (UID: \"c1a19051-1ac9-4caf-82df-5a004c49db68\") " pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:33:06.144993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.144971 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e78e94-cc94-496f-ac33-f4e708709d9a-metrics-certs\") pod \"router-default-76598c4567-mhntk\" (UID: \"f7e78e94-cc94-496f-ac33-f4e708709d9a\") " pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:06.144993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.144968 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"image-registry-7947ff7c95-mx6lt\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:06.145278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.145030 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tjjj9\" (UID: \"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:33:06.145278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.145216 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff6d432e-6834-4230-9a45-57209c333b93-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l2nk6\" (UID: \"ff6d432e-6834-4230-9a45-57209c333b93\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:33:06.146050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.146032 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0009777c-b015-471c-8d88-72d341de5778-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9lmn\" (UID: \"0009777c-b015-471c-8d88-72d341de5778\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:33:06.146822 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.146803 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ece428ad-e7c3-4744-925a-036290ed726e-metrics-tls\") pod \"dns-default-h9cmn\" (UID: \"ece428ad-e7c3-4744-925a-036290ed726e\") " pod="openshift-dns/dns-default-h9cmn" Nov 04 02:33:06.358129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.358089 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" Nov 04 02:33:06.359217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.359177 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:06.359356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.359275 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:06.359356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.359329 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" Nov 04 02:33:06.363508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.363493 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:06.363600 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.363545 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" Nov 04 02:33:06.368645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.368627 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbppk" Nov 04 02:33:06.370781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.370768 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9cmn" Nov 04 02:33:06.371878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.371860 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hl5s9" Nov 04 02:33:06.795712 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.795647 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9"] Nov 04 02:33:06.797971 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.797943 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c9ee7b_40be_4ecd_8459_bda33ae4fa2f.slice/crio-eac00707a22d55610c67d4d6294f226dbbe866137c1f340f3a1da02a37f7a6a0 WatchSource:0}: Error finding container eac00707a22d55610c67d4d6294f226dbbe866137c1f340f3a1da02a37f7a6a0: Status 404 returned error can't find the container with id eac00707a22d55610c67d4d6294f226dbbe866137c1f340f3a1da02a37f7a6a0 Nov 04 02:33:06.846279 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.846244 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3264410_13ad_450b_9c9a_3d2583359b09.slice/crio-c7a14aed3d4d65f951810fd6cc94fe01e10c8bbc8b54d5b4e2702108c435d755 WatchSource:0}: Error finding container c7a14aed3d4d65f951810fd6cc94fe01e10c8bbc8b54d5b4e2702108c435d755: Status 404 returned error can't find the container with id c7a14aed3d4d65f951810fd6cc94fe01e10c8bbc8b54d5b4e2702108c435d755 Nov 04 02:33:06.847203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.846966 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:33:06.847740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.847720 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hl5s9"] Nov 04 02:33:06.849289 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.849269 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a19051_1ac9_4caf_82df_5a004c49db68.slice/crio-ad761a976f2801b9546f19d1ec7a42ed50e404598f66b4a95744d5e2fcdd133e WatchSource:0}: Error finding container ad761a976f2801b9546f19d1ec7a42ed50e404598f66b4a95744d5e2fcdd133e: Status 404 returned error can't find the container with id ad761a976f2801b9546f19d1ec7a42ed50e404598f66b4a95744d5e2fcdd133e Nov 04 02:33:06.895612 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.895571 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8071b266_034b_4622_822b_f8b01b65a685.slice/crio-e27366914e1a807e88380772f9b0afb1986a3935c6ccb311dff05d0b505d5182 WatchSource:0}: Error finding container e27366914e1a807e88380772f9b0afb1986a3935c6ccb311dff05d0b505d5182: Status 404 returned error can't find the container with id e27366914e1a807e88380772f9b0afb1986a3935c6ccb311dff05d0b505d5182 Nov 04 02:33:06.897845 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.897824 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e78e94_cc94_496f_ac33_f4e708709d9a.slice/crio-9ac5d01c74df4a81a7ca038da8024e115b0b5f8f5bd69d1c22dd6f1454a5dc26 WatchSource:0}: Error finding container 9ac5d01c74df4a81a7ca038da8024e115b0b5f8f5bd69d1c22dd6f1454a5dc26: Status 404 returned error can't find the container with id 9ac5d01c74df4a81a7ca038da8024e115b0b5f8f5bd69d1c22dd6f1454a5dc26 Nov 04 02:33:06.898331 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.898314 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76598c4567-mhntk"] Nov 04 02:33:06.899020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.898997 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:33:06.912619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.909546 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l2nk6"] Nov 04 02:33:06.912619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.911688 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbppk"] Nov 04 02:33:06.917134 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.917074 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220ee58a_5ca9_4f09_a317_e5b66524efb0.slice/crio-0274e0a336b1f43ed3178ca941ef178b92774a2c1e62ef7071d616f600e6f722 WatchSource:0}: Error finding container 0274e0a336b1f43ed3178ca941ef178b92774a2c1e62ef7071d616f600e6f722: Status 404 returned error can't find the container with id 0274e0a336b1f43ed3178ca941ef178b92774a2c1e62ef7071d616f600e6f722 Nov 04 02:33:06.918216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.918137 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9cmn"] Nov 04 02:33:06.925686 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:06.925603 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece428ad_e7c3_4744_925a_036290ed726e.slice/crio-a35551d21c08fbfc034f5b3a0bca124836f3368c67e643fb42958c2de2fc2572 WatchSource:0}: Error finding container a35551d21c08fbfc034f5b3a0bca124836f3368c67e643fb42958c2de2fc2572: Status 404 returned error can't find the container with id a35551d21c08fbfc034f5b3a0bca124836f3368c67e643fb42958c2de2fc2572 Nov 04 02:33:06.956891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:06.956014 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn"] Nov 04 02:33:07.329170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.329134 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" event={"ID":"ff6d432e-6834-4230-9a45-57209c333b93","Type":"ContainerStarted","Data":"8e2153eeda551c014c686a50aa34ec9c29e8b66491348e09db29993d99765fc1"} Nov 04 02:33:07.330824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.330751 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbppk" event={"ID":"220ee58a-5ca9-4f09-a317-e5b66524efb0","Type":"ContainerStarted","Data":"0274e0a336b1f43ed3178ca941ef178b92774a2c1e62ef7071d616f600e6f722"} Nov 04 02:33:07.332320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.332292 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" event={"ID":"0009777c-b015-471c-8d88-72d341de5778","Type":"ContainerStarted","Data":"8dbc422b16f3b92fab8ac0fd488286e12222168a43234c59d55fe591231b20b6"} Nov 04 02:33:07.333970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.333948 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl5s9" event={"ID":"c1a19051-1ac9-4caf-82df-5a004c49db68","Type":"ContainerStarted","Data":"41a2ba13efa15ef62ccce146a8b18fd1e413c6cfc8e81d8ac002d7dcdaf0d3ce"} Nov 04 02:33:07.334062 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.333988 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl5s9" event={"ID":"c1a19051-1ac9-4caf-82df-5a004c49db68","Type":"ContainerStarted","Data":"ad761a976f2801b9546f19d1ec7a42ed50e404598f66b4a95744d5e2fcdd133e"} Nov 04 02:33:07.335569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.335548 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76598c4567-mhntk" event={"ID":"f7e78e94-cc94-496f-ac33-f4e708709d9a","Type":"ContainerStarted","Data":"297bc49aabd93f9739ddcd3f373dd75364f6ec0f87a9cebd5d7bd300bf553076"} Nov 04 02:33:07.335665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.335581 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76598c4567-mhntk" event={"ID":"f7e78e94-cc94-496f-ac33-f4e708709d9a","Type":"ContainerStarted","Data":"9ac5d01c74df4a81a7ca038da8024e115b0b5f8f5bd69d1c22dd6f1454a5dc26"} Nov 04 02:33:07.337105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.337084 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" event={"ID":"8071b266-034b-4622-822b-f8b01b65a685","Type":"ContainerStarted","Data":"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc"} Nov 04 02:33:07.337208 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.337116 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" event={"ID":"8071b266-034b-4622-822b-f8b01b65a685","Type":"ContainerStarted","Data":"e27366914e1a807e88380772f9b0afb1986a3935c6ccb311dff05d0b505d5182"} Nov 04 02:33:07.337208 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.337135 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:07.338486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.338466 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" event={"ID":"f3264410-13ad-450b-9c9a-3d2583359b09","Type":"ContainerStarted","Data":"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14"} Nov 04 02:33:07.338580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.338498 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" event={"ID":"f3264410-13ad-450b-9c9a-3d2583359b09","Type":"ContainerStarted","Data":"c7a14aed3d4d65f951810fd6cc94fe01e10c8bbc8b54d5b4e2702108c435d755"} Nov 04 02:33:07.338580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.338515 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:07.339640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.339619 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" event={"ID":"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f","Type":"ContainerStarted","Data":"eac00707a22d55610c67d4d6294f226dbbe866137c1f340f3a1da02a37f7a6a0"} Nov 04 02:33:07.340627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.340607 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9cmn" event={"ID":"ece428ad-e7c3-4744-925a-036290ed726e","Type":"ContainerStarted","Data":"a35551d21c08fbfc034f5b3a0bca124836f3368c67e643fb42958c2de2fc2572"} Nov 04 02:33:07.360324 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.360299 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:07.362962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.362935 4784 patch_prober.go:29] interesting pod/router-default-76598c4567-mhntk container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 04 02:33:07.362962 ip-10-0-133-209 kubenswrapper[4784]: [-]has-synced failed: reason withheld Nov 04 02:33:07.362962 ip-10-0-133-209 kubenswrapper[4784]: [+]process-running ok Nov 04 02:33:07.362962 ip-10-0-133-209 kubenswrapper[4784]: healthz check failed Nov 04 02:33:07.363132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:07.362982 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-76598c4567-mhntk" podUID="f7e78e94-cc94-496f-ac33-f4e708709d9a" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 04 02:33:08.153343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.153312 4784 scope.go:122] "RemoveContainer" containerID="a89e3b8711bdb2f734159cff5b0b5aafdd2c69360e0791dd9bab44f9defb9e1b" Nov 04 02:33:08.348249 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.348218 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:33:08.348735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.348393 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" event={"ID":"de579720-94f5-4cbd-b9f6-60e5b3d82647","Type":"ContainerStarted","Data":"1d73d651a9e5041ee08417d0070f3af5dd3cfe4dee21f85bcf59ebac4a5153da"} Nov 04 02:33:08.349135 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.349096 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:33:08.350389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.350368 4784 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zp9vk container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/readyz\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Nov 04 02:33:08.350506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.350407 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" podUID="de579720-94f5-4cbd-b9f6-60e5b3d82647" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.15:8443/readyz\": dial tcp 10.133.0.15:8443: connect: connection refused" Nov 04 02:33:08.361018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:08.360992 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:09.119785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.119202 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:33:09.358871 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.358829 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl5s9" event={"ID":"c1a19051-1ac9-4caf-82df-5a004c49db68","Type":"ContainerStarted","Data":"3e366d666ba644e24fe3868bee21c76cd29f4a707d43a11859d0865768f13f38"} Nov 04 02:33:09.360930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.360902 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:09.362979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.362959 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76598c4567-mhntk" Nov 04 02:33:09.367066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.366503 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zp9vk" Nov 04 02:33:09.983428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:09.983389 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xfw5" Nov 04 02:33:10.370524 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:10.370489 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-l2nk6" event={"ID":"ff6d432e-6834-4230-9a45-57209c333b93","Type":"ContainerStarted","Data":"b6d17df0c8380b2d68f6e273e2a2de9bebcf7d0bd5ec3df3dd1e80b205498193"} Nov 04 02:33:10.373964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:10.373715 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbppk" event={"ID":"220ee58a-5ca9-4f09-a317-e5b66524efb0","Type":"ContainerStarted","Data":"bcedb28309f9ed1588000c40b813422f7a4fc4b10771e3c8c54ccfbc9388090b"} Nov 04 02:33:10.373964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:10.373768 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbppk" event={"ID":"220ee58a-5ca9-4f09-a317-e5b66524efb0","Type":"ContainerStarted","Data":"00fe36f90ae0da61bdce1bae0eebb60673497f27f12c9f121f39ad7baf05affa"} Nov 04 02:33:11.380362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.380307 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" event={"ID":"0009777c-b015-471c-8d88-72d341de5778","Type":"ContainerStarted","Data":"94dcd5c78cb06588c0b65b43fac3a66ff52beae01464f84f9f1dcdcee37e9e6e"} Nov 04 02:33:11.380861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.380377 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9lmn" event={"ID":"0009777c-b015-471c-8d88-72d341de5778","Type":"ContainerStarted","Data":"7db1c1c0c26a92e41776180610b06bff1a4000e935039710d173e23a4fb6e748"} Nov 04 02:33:11.382164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.382123 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tjjj9" event={"ID":"c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f","Type":"ContainerStarted","Data":"2e97d474b98b21598355fb0cbbc39c7272cc31b5dd1391b87c15a4e49356e055"} Nov 04 02:33:11.384978 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.384947 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9cmn" event={"ID":"ece428ad-e7c3-4744-925a-036290ed726e","Type":"ContainerStarted","Data":"75cd1d0cda2ced53fc8b838188944ca700978a42458b30ae1778b86590892ba2"} Nov 04 02:33:11.385082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.384990 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9cmn" event={"ID":"ece428ad-e7c3-4744-925a-036290ed726e","Type":"ContainerStarted","Data":"3124cfeadfc65ef636d86b2ec7548636b20f29d584c9091bf179b765fdd88cc1"} Nov 04 02:33:11.385082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:11.385011 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h9cmn" Nov 04 02:33:12.391954 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:12.391862 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl5s9" event={"ID":"c1a19051-1ac9-4caf-82df-5a004c49db68","Type":"ContainerStarted","Data":"f7b40dbe70e76b153684c2b671ef4e4472951cdfb6da48309c983e2a553d45f6"} Nov 04 02:33:21.395066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:21.395026 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h9cmn" Nov 04 02:33:26.115501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.115462 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pj2n2"] Nov 04 02:33:26.118864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.118847 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.172182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.172139 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 02:33:26.213183 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213138 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-root\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213221 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-sys\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213290 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213339 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213445 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqck\" (UniqueName: \"kubernetes.io/projected/3aa62e5d-17b8-4174-8aba-200941de3f05-kube-api-access-2pqck\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213478 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-tls\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213508 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-wtmp\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213580 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-metrics-client-ca\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.213695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.213609 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-textfile\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.223655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.223624 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 02:33:26.223655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.223624 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 02:33:26.223838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.223656 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 02:33:26.223838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.223677 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mpgnq\"" Nov 04 02:33:26.314421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314385 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314427 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqck\" (UniqueName: \"kubernetes.io/projected/3aa62e5d-17b8-4174-8aba-200941de3f05-kube-api-access-2pqck\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314453 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-tls\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314479 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-wtmp\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314503 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-metrics-client-ca\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314525 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-textfile\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314552 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-root\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314649 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-root\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314660 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-wtmp\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.314702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314589 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-sys\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.315131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314713 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.315131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314777 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa62e5d-17b8-4174-8aba-200941de3f05-sys\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.315131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.314956 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-textfile\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.315131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.315085 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-metrics-client-ca\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.315326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.315131 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.317015 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.316996 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.317238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.317221 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3aa62e5d-17b8-4174-8aba-200941de3f05-node-exporter-tls\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.368049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.367965 4784 patch_prober.go:29] interesting pod/image-registry-7b7c9cfd78-tdjrx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 02:33:26.368049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.368030 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" podUID="8071b266-034b-4622-822b-f8b01b65a685" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:33:26.376722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.376690 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqck\" (UniqueName: \"kubernetes.io/projected/3aa62e5d-17b8-4174-8aba-200941de3f05-kube-api-access-2pqck\") pod \"node-exporter-pj2n2\" (UID: \"3aa62e5d-17b8-4174-8aba-200941de3f05\") " pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.430562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:26.430523 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pj2n2" Nov 04 02:33:26.441221 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:33:26.441159 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa62e5d_17b8_4174_8aba_200941de3f05.slice/crio-3ff3ed3b6789653efef91f30ecb3dfcfc4ee3ef0136ea4dfe0402d23a6162604 WatchSource:0}: Error finding container 3ff3ed3b6789653efef91f30ecb3dfcfc4ee3ef0136ea4dfe0402d23a6162604: Status 404 returned error can't find the container with id 3ff3ed3b6789653efef91f30ecb3dfcfc4ee3ef0136ea4dfe0402d23a6162604 Nov 04 02:33:27.449515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:27.449474 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj2n2" event={"ID":"3aa62e5d-17b8-4174-8aba-200941de3f05","Type":"ContainerStarted","Data":"3ff3ed3b6789653efef91f30ecb3dfcfc4ee3ef0136ea4dfe0402d23a6162604"} Nov 04 02:33:28.355064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:28.355031 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:29.367099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:29.367070 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:29.459347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:29.459310 4784 generic.go:356] "Generic (PLEG): container finished" podID="3aa62e5d-17b8-4174-8aba-200941de3f05" containerID="2f7dc9908c789f9b809289dd188fea554fb910c61755557043a6a1330c0a3c37" exitCode=0 Nov 04 02:33:29.459528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:29.459384 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj2n2" event={"ID":"3aa62e5d-17b8-4174-8aba-200941de3f05","Type":"ContainerDied","Data":"2f7dc9908c789f9b809289dd188fea554fb910c61755557043a6a1330c0a3c37"} Nov 04 02:33:30.465948 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:30.465917 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj2n2" event={"ID":"3aa62e5d-17b8-4174-8aba-200941de3f05","Type":"ContainerStarted","Data":"e77f387b417506355ddfeae94358d972cd168b7bd9fd0f93526abb2f5ddb61d1"} Nov 04 02:33:30.466386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:30.465966 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj2n2" event={"ID":"3aa62e5d-17b8-4174-8aba-200941de3f05","Type":"ContainerStarted","Data":"ba62ebd5040d682f5b59e300af75128a7c231fd2a604b77d3902e965de7111aa"} Nov 04 02:33:30.587601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:30.587545 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pj2n2" podStartSLOduration=2.329130458 podStartE2EDuration="4.58753021s" totalImagesPullingTime="2.18186612s" totalInitContainerRuntime="76.533632ms" isStatefulPod=true podCreationTimestamp="2026-11-04 02:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:30.586938138 +0000 UTC m=+57.553439130" watchObservedRunningTime="2026-11-04 02:33:30.58753021 +0000 UTC m=+57.554031215" Nov 04 02:33:33.013668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:33.013637 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:33:34.394370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.394325 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" podUID="f3264410-13ad-450b-9c9a-3d2583359b09" containerName="registry" containerID="cri-o://4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14" gracePeriod=30 Nov 04 02:33:34.621368 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.621345 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:34.683862 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.683831 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.683889 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.683909 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.683960 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.683977 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.684016 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.684049 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.684292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.684081 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") pod \"f3264410-13ad-450b-9c9a-3d2583359b09\" (UID: \"f3264410-13ad-450b-9c9a-3d2583359b09\") " Nov 04 02:33:34.685017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.684677 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:34.685149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.685035 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:34.687479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.687449 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:33:34.687749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.687722 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:34.687890 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.687867 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:33:34.687951 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.687919 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "kube-api-access-ctdbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:34.687993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.687966 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:34.692030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.692009 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted" pod "f3264410-13ad-450b-9c9a-3d2583359b09" (UID: "f3264410-13ad-450b-9c9a-3d2583359b09"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:33:34.784845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784811 4784 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3264410-13ad-450b-9c9a-3d2583359b09-ca-trust-extracted\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.784845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784839 4784 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-registry-tls\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.784845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784848 4784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-trusted-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.785073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784858 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdbg\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-kube-api-access-ctdbg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.785073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784869 4784 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3264410-13ad-450b-9c9a-3d2583359b09-registry-certificates\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.785073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784877 4784 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3264410-13ad-450b-9c9a-3d2583359b09-bound-sa-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.785073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784887 4784 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-installation-pull-secrets\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:34.785073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:34.784897 4784 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3264410-13ad-450b-9c9a-3d2583359b09-image-registry-private-configuration\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:35.486391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.486305 4784 generic.go:356] "Generic (PLEG): container finished" podID="f3264410-13ad-450b-9c9a-3d2583359b09" containerID="4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14" exitCode=0 Nov 04 02:33:35.486391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.486355 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" Nov 04 02:33:35.486391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.486369 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" event={"ID":"f3264410-13ad-450b-9c9a-3d2583359b09","Type":"ContainerDied","Data":"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14"} Nov 04 02:33:35.486864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.486411 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7947ff7c95-mx6lt" event={"ID":"f3264410-13ad-450b-9c9a-3d2583359b09","Type":"ContainerDied","Data":"c7a14aed3d4d65f951810fd6cc94fe01e10c8bbc8b54d5b4e2702108c435d755"} Nov 04 02:33:35.486864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.486431 4784 scope.go:122] "RemoveContainer" containerID="4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14" Nov 04 02:33:35.495945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.495869 4784 scope.go:122] "RemoveContainer" containerID="4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14" Nov 04 02:33:35.496496 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:33:35.496474 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14\": container with ID starting with 4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14 not found: ID does not exist" containerID="4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14" Nov 04 02:33:35.496585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.496507 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14"} err="failed to get container status \"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14\": rpc error: code = NotFound desc = could not find container \"4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14\": container with ID starting with 4a4cd78a9d300b0132fc089147f7ea0094de803e5af893c959ec2b9a253eea14 not found: ID does not exist" Nov 04 02:33:35.653246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.653209 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:33:35.704231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:35.704175 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7947ff7c95-mx6lt"] Nov 04 02:33:36.398565 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:33:36.398528 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:33:37.157838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:37.157805 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3264410-13ad-450b-9c9a-3d2583359b09" path="/var/lib/kubelet/pods/f3264410-13ad-450b-9c9a-3d2583359b09/volumes" Nov 04 02:33:54.559495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:54.559449 4784 generic.go:356] "Generic (PLEG): container finished" podID="3ea891a3-ac68-4f11-add3-b42a1b657846" containerID="aac337142b53e93f28d60efccae61cfd7e147d7826f5049b25e1d17002bb992f" exitCode=0 Nov 04 02:33:54.559934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:54.559514 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" event={"ID":"3ea891a3-ac68-4f11-add3-b42a1b657846","Type":"ContainerDied","Data":"aac337142b53e93f28d60efccae61cfd7e147d7826f5049b25e1d17002bb992f"} Nov 04 02:33:54.559934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:54.559832 4784 scope.go:122] "RemoveContainer" containerID="aac337142b53e93f28d60efccae61cfd7e147d7826f5049b25e1d17002bb992f" Nov 04 02:33:55.564532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:55.564500 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nxqpl" event={"ID":"3ea891a3-ac68-4f11-add3-b42a1b657846","Type":"ContainerStarted","Data":"db2b721638370a2b42f54352fabb2d164a9aa5ac5609cbe32cfd58bad9576ef8"} Nov 04 02:33:58.033467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.033403 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" podUID="8071b266-034b-4622-822b-f8b01b65a685" containerName="registry" containerID="cri-o://84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc" gracePeriod=30 Nov 04 02:33:58.286912 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.286853 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:58.388447 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388392 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388470 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388508 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388534 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388577 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388604 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388637 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.388914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.388665 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") pod \"8071b266-034b-4622-822b-f8b01b65a685\" (UID: \"8071b266-034b-4622-822b-f8b01b65a685\") " Nov 04 02:33:58.389068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.389027 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:58.389412 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.389383 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:58.391047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.391013 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:33:58.391536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.391494 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:33:58.391617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.391523 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:58.391617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.391564 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "kube-api-access-gnslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:58.392369 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.392349 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:58.395762 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.395738 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted" pod "8071b266-034b-4622-822b-f8b01b65a685" (UID: "8071b266-034b-4622-822b-f8b01b65a685"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:33:58.489701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489663 4784 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-installation-pull-secrets\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489700 4784 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-registry-tls\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489710 4784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-trusted-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489719 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnslq\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-kube-api-access-gnslq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489728 4784 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8071b266-034b-4622-822b-f8b01b65a685-bound-sa-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489739 4784 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8071b266-034b-4622-822b-f8b01b65a685-image-registry-private-configuration\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489749 4784 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8071b266-034b-4622-822b-f8b01b65a685-registry-certificates\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.489929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.489758 4784 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8071b266-034b-4622-822b-f8b01b65a685-ca-trust-extracted\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:58.580981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.580900 4784 generic.go:356] "Generic (PLEG): container finished" podID="8071b266-034b-4622-822b-f8b01b65a685" containerID="84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc" exitCode=0 Nov 04 02:33:58.580981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.580945 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" Nov 04 02:33:58.581237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.580985 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" event={"ID":"8071b266-034b-4622-822b-f8b01b65a685","Type":"ContainerDied","Data":"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc"} Nov 04 02:33:58.581237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.581042 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7c9cfd78-tdjrx" event={"ID":"8071b266-034b-4622-822b-f8b01b65a685","Type":"ContainerDied","Data":"e27366914e1a807e88380772f9b0afb1986a3935c6ccb311dff05d0b505d5182"} Nov 04 02:33:58.581237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.581071 4784 scope.go:122] "RemoveContainer" containerID="84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc" Nov 04 02:33:58.590262 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.590240 4784 scope.go:122] "RemoveContainer" containerID="84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc" Nov 04 02:33:58.590543 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:33:58.590522 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc\": container with ID starting with 84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc not found: ID does not exist" containerID="84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc" Nov 04 02:33:58.590585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.590553 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc"} err="failed to get container status \"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc\": rpc error: code = NotFound desc = could not find container \"84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc\": container with ID starting with 84596e90e8d73bf643501b72fffbb85ef1b7f68c96b97b1fda6eee0862ce6dbc not found: ID does not exist" Nov 04 02:33:58.751794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.751761 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:33:58.755882 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:58.755853 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b7c9cfd78-tdjrx"] Nov 04 02:33:59.157701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:33:59.157670 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8071b266-034b-4622-822b-f8b01b65a685" path="/var/lib/kubelet/pods/8071b266-034b-4622-822b-f8b01b65a685/volumes" Nov 04 02:34:05.608768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:05.608738 4784 generic.go:356] "Generic (PLEG): container finished" podID="a06b3575-3fcd-4564-89b5-c7822dc36b59" containerID="058fb1ea720add341b4db63342d1649f0f67a47fa6459d20a2852b067c666321" exitCode=0 Nov 04 02:34:05.609174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:05.608781 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" event={"ID":"a06b3575-3fcd-4564-89b5-c7822dc36b59","Type":"ContainerDied","Data":"058fb1ea720add341b4db63342d1649f0f67a47fa6459d20a2852b067c666321"} Nov 04 02:34:05.609174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:05.609065 4784 scope.go:122] "RemoveContainer" containerID="058fb1ea720add341b4db63342d1649f0f67a47fa6459d20a2852b067c666321" Nov 04 02:34:06.399130 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:34:06.399079 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:34:06.615659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:06.615562 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vtfv2" event={"ID":"a06b3575-3fcd-4564-89b5-c7822dc36b59","Type":"ContainerStarted","Data":"550c40f5ac998f7bb0e41e80f34b51d8dbc3070b34e3c20bd6950dd96d7c7a6c"} Nov 04 02:34:06.616982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:06.616956 4784 generic.go:356] "Generic (PLEG): container finished" podID="b1bb9090-5872-40b1-ae31-b910dc6575fa" containerID="7de19416eb5e6e4670dee89edf0be7bf150ec36fa5b2e3fe6ff5e7ec847293e9" exitCode=0 Nov 04 02:34:06.617111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:06.617031 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" event={"ID":"b1bb9090-5872-40b1-ae31-b910dc6575fa","Type":"ContainerDied","Data":"7de19416eb5e6e4670dee89edf0be7bf150ec36fa5b2e3fe6ff5e7ec847293e9"} Nov 04 02:34:06.617351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:06.617336 4784 scope.go:122] "RemoveContainer" containerID="7de19416eb5e6e4670dee89edf0be7bf150ec36fa5b2e3fe6ff5e7ec847293e9" Nov 04 02:34:07.623370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:34:07.623337 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tbjj2" event={"ID":"b1bb9090-5872-40b1-ae31-b910dc6575fa","Type":"ContainerStarted","Data":"90d370d56f4032fa2506d118fdcdd6554dfe385cb622e246e6008ec4fcf52ef5"} Nov 04 02:34:36.400729 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:34:36.400677 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:35:06.399311 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:35:06.399265 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:35:36.399544 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:35:36.399493 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:36:06.399081 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:36:06.399038 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:36:36.398223 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:36:36.398160 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:37:06.399157 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:37:06.399121 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:37:22.291495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291459 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4"] Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291825 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f3264410-13ad-450b-9c9a-3d2583359b09" containerName="registry" Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291838 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3264410-13ad-450b-9c9a-3d2583359b09" containerName="registry" Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291850 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8071b266-034b-4622-822b-f8b01b65a685" containerName="registry" Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291855 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8071b266-034b-4622-822b-f8b01b65a685" containerName="registry" Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291907 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3264410-13ad-450b-9c9a-3d2583359b09" containerName="registry" Nov 04 02:37:22.292018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.291915 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8071b266-034b-4622-822b-f8b01b65a685" containerName="registry" Nov 04 02:37:22.294938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.294918 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.308944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.308917 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8vz\" (UniqueName: \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.309058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.308963 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.309105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.309066 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.348796 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.348765 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 02:37:22.399850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.399818 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 02:37:22.400092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.400074 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-m6w89\"" Nov 04 02:37:22.401037 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.401018 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4"] Nov 04 02:37:22.410217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.410178 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.410347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.410263 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.410347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.410290 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8vz\" (UniqueName: \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.410563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.410548 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.410610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.410581 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.473204 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.473169 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8vz\" (UniqueName: \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.610231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.610125 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:37:22.985102 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:22.985072 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4"] Nov 04 02:37:22.988005 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:22.987974 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338ec91f_494a_405e_b297_a758f1ee79e3.slice/crio-b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248 WatchSource:0}: Error finding container b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248: Status 404 returned error can't find the container with id b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248 Nov 04 02:37:23.304647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.304560 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerStarted","Data":"b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248"} Nov 04 02:37:23.551512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.551475 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh"] Nov 04 02:37:23.556066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.556001 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.618715 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.618674 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.618873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.618802 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdvg\" (UniqueName: \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.618873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.618841 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.663442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.663411 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dtptm\"" Nov 04 02:37:23.664483 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.664459 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh"] Nov 04 02:37:23.665162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.665143 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw"] Nov 04 02:37:23.668562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.668548 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.719131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719100 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.719374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719143 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.719374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719165 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.719374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719222 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.719374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719275 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486w2\" (UniqueName: \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.719612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719389 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdvg\" (UniqueName: \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.719612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719586 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.719681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.719583 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.722673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.722647 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw"] Nov 04 02:37:23.723090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.723074 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-vmkvg\"" Nov 04 02:37:23.763269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.763228 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m"] Nov 04 02:37:23.766663 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.766648 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.781987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.781965 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdvg\" (UniqueName: \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.820014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.819933 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486w2\" (UniqueName: \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.820014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.819979 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hrc\" (UniqueName: \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.820014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820006 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.820280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820144 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.820280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820212 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.820360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820323 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.820579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820562 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.820617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.820590 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.823152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.823122 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wtk6r\"" Nov 04 02:37:23.824238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.824215 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m"] Nov 04 02:37:23.859123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.859086 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9"] Nov 04 02:37:23.862891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.862868 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:23.871641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.871622 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:23.883294 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.883271 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486w2\" (UniqueName: \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.920942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.920897 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltx5\" (UniqueName: \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:23.921157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.920991 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hrc\" (UniqueName: \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.921157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921037 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.921157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921067 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:23.921157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921117 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9"] Nov 04 02:37:23.921474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921213 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.921533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921502 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:23.921591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921531 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.921591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.921582 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:23.924014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.923990 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-bsbnz\"" Nov 04 02:37:23.983699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.983665 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:23.984099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:23.984074 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hrc\" (UniqueName: \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:24.022427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.022384 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.022427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.022433 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.023496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.022452 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltx5\" (UniqueName: \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.023496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.023010 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.023496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.023173 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.082711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.082627 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:24.085499 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.085479 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltx5\" (UniqueName: \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.181419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.181378 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:37:24.254540 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.254511 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh"] Nov 04 02:37:24.259229 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:24.259168 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef0c230_1c45_47ed_89a3_bfc87b0b57fb.slice/crio-abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0 WatchSource:0}: Error finding container abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0: Status 404 returned error can't find the container with id abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0 Nov 04 02:37:24.309571 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.309535 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" event={"ID":"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb","Type":"ContainerStarted","Data":"abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0"} Nov 04 02:37:24.360928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.360903 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw"] Nov 04 02:37:24.363155 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:24.363122 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cbd183_02fa_4b53_b13b_7f8eaf2095af.slice/crio-df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388 WatchSource:0}: Error finding container df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388: Status 404 returned error can't find the container with id df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388 Nov 04 02:37:24.460805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.460781 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m"] Nov 04 02:37:24.463223 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:24.463181 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c38ea37_f28b_4f84_b86f_cd3f5f0b1419.slice/crio-93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42 WatchSource:0}: Error finding container 93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42: Status 404 returned error can't find the container with id 93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42 Nov 04 02:37:24.559604 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:24.559569 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9"] Nov 04 02:37:24.562748 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:24.562719 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f974998_5941_487b_bd70_f012ab01906f.slice/crio-a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a WatchSource:0}: Error finding container a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a: Status 404 returned error can't find the container with id a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a Nov 04 02:37:25.249593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.249554 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf"] Nov 04 02:37:25.253378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.253345 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.307041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.307007 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-446w6\"" Nov 04 02:37:25.322665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.322629 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" event={"ID":"0f974998-5941-487b-bd70-f012ab01906f","Type":"ContainerStarted","Data":"a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a"} Nov 04 02:37:25.325580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.325550 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerStarted","Data":"df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388"} Nov 04 02:37:25.327363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.327335 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerStarted","Data":"93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42"} Nov 04 02:37:25.337504 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.337467 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.337647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.337520 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.337647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.337577 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fc8s\" (UniqueName: \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.359315 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.359279 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf"] Nov 04 02:37:25.438975 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.438934 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.439163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.438992 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.439163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.439050 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fc8s\" (UniqueName: \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.439733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.439713 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.439925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.439907 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.513667 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.513562 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fc8s\" (UniqueName: \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.573421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.573092 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:37:25.831407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.831322 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf"] Nov 04 02:37:25.836892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.836853 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.842040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.842015 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.842177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.842060 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.842177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.842112 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29s42\" (UniqueName: \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.892830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.892121 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sxdnd\"" Nov 04 02:37:25.893642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.893617 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf"] Nov 04 02:37:25.943870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.943414 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.943870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.943476 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.943870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.943536 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29s42\" (UniqueName: \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.943870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.943824 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:25.944232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:25.944096 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:26.019606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:26.019574 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29s42\" (UniqueName: \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:26.071374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:26.071224 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf"] Nov 04 02:37:26.156588 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:26.156541 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:37:26.338488 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:26.338277 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerStarted","Data":"dd1924d7e7359215efc88d1002a51f0283c885593ef15e864dbe90f15cd7493d"} Nov 04 02:37:26.566802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:26.566729 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf"] Nov 04 02:37:27.354995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.354939 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerStarted","Data":"17c7d80eb87ca509e9887b3e62bbf687a7833527b7811b00e9c380f099449360"} Nov 04 02:37:27.394226 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.393533 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 02:37:27.398235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.397846 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.451811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.451773 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 02:37:27.460028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.459996 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7m6\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460090 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460155 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460309 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460411 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460357 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460411 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460390 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460435 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460458 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.460519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.460483 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.476247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.476218 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pm6wh\"" Nov 04 02:37:27.504962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.504929 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 02:37:27.561631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561594 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.561808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561659 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.561808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561701 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.561808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561744 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.561808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561769 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.561808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561793 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.562082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7m6\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.562082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561895 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.562082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.561928 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.562885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.562851 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.563867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.563843 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.564280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.564255 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.565130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.565107 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.565476 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.565442 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.572972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.572889 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.573335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.573259 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.639504 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.639424 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.642814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.642433 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7m6\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\") pod \"maas-default-gateway-openshift-default-76597d6855-zhmt8\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.708231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.706303 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2"] Nov 04 02:37:27.712567 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.711559 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.714757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.714573 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:27.763717 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763680 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kmz\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-kube-api-access-44kmz\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763772 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08528d88-4550-4f16-8e49-24e22757f3d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763830 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763855 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763899 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08528d88-4550-4f16-8e49-24e22757f3d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763936 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763961 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.763999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.763985 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.764176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.764011 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.770151 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.770122 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hc672\"" Nov 04 02:37:27.775360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.772327 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2"] Nov 04 02:37:27.864920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.864858 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kmz\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-kube-api-access-44kmz\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.864988 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08528d88-4550-4f16-8e49-24e22757f3d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865047 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865077 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865155 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08528d88-4550-4f16-8e49-24e22757f3d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865213 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865240 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865280 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.865393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.865305 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.866873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.866844 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08528d88-4550-4f16-8e49-24e22757f3d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.867516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.867495 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.867767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.867728 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.868430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.868408 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.868565 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.868521 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.871882 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.871847 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08528d88-4550-4f16-8e49-24e22757f3d7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.874105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.874082 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08528d88-4550-4f16-8e49-24e22757f3d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.945352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.945321 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kmz\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-kube-api-access-44kmz\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:27.948869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:27.948844 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08528d88-4550-4f16-8e49-24e22757f3d7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2\" (UID: \"08528d88-4550-4f16-8e49-24e22757f3d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:28.031848 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:28.031600 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:28.201217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:28.201065 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 02:37:28.211214 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:37:28.211162 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1afcda6_7342_4238_a55b_14b7aee67b61.slice/crio-5ead32480ca9ca74f1295c39926a42eb1574c9ca6ae028f12eb94a37600ea4ee WatchSource:0}: Error finding container 5ead32480ca9ca74f1295c39926a42eb1574c9ca6ae028f12eb94a37600ea4ee: Status 404 returned error can't find the container with id 5ead32480ca9ca74f1295c39926a42eb1574c9ca6ae028f12eb94a37600ea4ee Nov 04 02:37:28.368175 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:28.368136 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerStarted","Data":"5ead32480ca9ca74f1295c39926a42eb1574c9ca6ae028f12eb94a37600ea4ee"} Nov 04 02:37:28.466667 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:28.466588 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2"] Nov 04 02:37:29.389780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:29.389711 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" event={"ID":"08528d88-4550-4f16-8e49-24e22757f3d7","Type":"ContainerStarted","Data":"68862cca8a4c98c0d8239b791f094d5ad3a639b25d9896e20994b69856ea94a1"} Nov 04 02:37:33.231091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:33.231059 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:37:33.234085 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:33.234060 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:37:33.259732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:33.259617 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:37:33.275911 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:33.275834 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:37:36.420619 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:37:36.420536 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:37:37.892712 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.892676 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:37.893034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.892765 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:37.893034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.892805 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:37.900664 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.900624 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:37.900755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.900716 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:37.900796 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:37.900758 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:37:38.477852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:38.477783 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" event={"ID":"08528d88-4550-4f16-8e49-24e22757f3d7","Type":"ContainerStarted","Data":"9a5a634f92b3b59a0a4b444ea2879a9d7b02dcdbfc609a0e980bb265c6e2a2b7"} Nov 04 02:37:38.482825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:38.482229 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerStarted","Data":"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c"} Nov 04 02:37:38.706035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:38.705503 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" podStartSLOduration=2.284720006 podStartE2EDuration="11.705481437s" totalImagesPullingTime="9.420761431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:38.652955274 +0000 UTC m=+305.619456266" watchObservedRunningTime="2026-11-04 02:37:38.705481437 +0000 UTC m=+305.671982430" Nov 04 02:37:38.715175 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:38.715079 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:38.777617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:38.777494 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podStartSLOduration=2.100392718 podStartE2EDuration="11.777480073s" totalImagesPullingTime="9.677087355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:38.777261215 +0000 UTC m=+305.743762210" watchObservedRunningTime="2026-11-04 02:37:38.777480073 +0000 UTC m=+305.743981057" Nov 04 02:37:39.033964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.033928 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:39.040021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.039994 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:39.492755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.492712 4784 generic.go:356] "Generic (PLEG): container finished" podID="338ec91f-494a-405e-b297-a758f1ee79e3" containerID="0322cfd2cfd0d7b00c26c792072fb5ad8af988464ac86c274470ce4674068ba3" exitCode=0 Nov 04 02:37:39.492961 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.492841 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerDied","Data":"0322cfd2cfd0d7b00c26c792072fb5ad8af988464ac86c274470ce4674068ba3"} Nov 04 02:37:39.493394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.493375 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:39.494484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.494464 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2" Nov 04 02:37:39.494776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.494757 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:37:39.721967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:39.721936 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:40.499924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.499889 4784 generic.go:356] "Generic (PLEG): container finished" podID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerID="4afd3926eb46a116ec53dca730834fe8e226c124aa5b31c58f8a3090ffc1f3a6" exitCode=0 Nov 04 02:37:40.500386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.500004 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerDied","Data":"4afd3926eb46a116ec53dca730834fe8e226c124aa5b31c58f8a3090ffc1f3a6"} Nov 04 02:37:40.502363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.502333 4784 generic.go:356] "Generic (PLEG): container finished" podID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerID="bd6ea9f74ad7b6d60a15c09c4c5ae4615843cba73ed88abe7ef40f0e34c83039" exitCode=0 Nov 04 02:37:40.502496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.502469 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" event={"ID":"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb","Type":"ContainerDied","Data":"bd6ea9f74ad7b6d60a15c09c4c5ae4615843cba73ed88abe7ef40f0e34c83039"} Nov 04 02:37:40.505154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.505132 4784 generic.go:356] "Generic (PLEG): container finished" podID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerID="2292bedebed3ce9e93123fefb2c89fd315d8125cdbbff01056074423e49fce19" exitCode=0 Nov 04 02:37:40.506122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.506099 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerDied","Data":"2292bedebed3ce9e93123fefb2c89fd315d8125cdbbff01056074423e49fce19"} Nov 04 02:37:40.506520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.506418 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:40.507632 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:40.507621 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:37:41.512700 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:41.512664 4784 generic.go:356] "Generic (PLEG): container finished" podID="0f974998-5941-487b-bd70-f012ab01906f" containerID="501c6b9d24de78f4e35636ed8f189e8fc0fe5b3cdc481a8d33c1c398b844b43b" exitCode=0 Nov 04 02:37:41.513966 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:41.513942 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" event={"ID":"0f974998-5941-487b-bd70-f012ab01906f","Type":"ContainerDied","Data":"501c6b9d24de78f4e35636ed8f189e8fc0fe5b3cdc481a8d33c1c398b844b43b"} Nov 04 02:37:42.518090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.518059 4784 generic.go:356] "Generic (PLEG): container finished" podID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerID="d2d347d53f77cb08c6bb03a990582dd99eb4949c1d31b0e80a01468a6905a607" exitCode=0 Nov 04 02:37:42.518546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.518118 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerDied","Data":"d2d347d53f77cb08c6bb03a990582dd99eb4949c1d31b0e80a01468a6905a607"} Nov 04 02:37:42.519849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.519824 4784 generic.go:356] "Generic (PLEG): container finished" podID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerID="1eabbd820887e389586c66191bbbc043eac48632ab1c60beb62b888c1e336f97" exitCode=0 Nov 04 02:37:42.519981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.519896 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerDied","Data":"1eabbd820887e389586c66191bbbc043eac48632ab1c60beb62b888c1e336f97"} Nov 04 02:37:42.521721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.521623 4784 generic.go:356] "Generic (PLEG): container finished" podID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerID="13040b301aaa20e78d73d484f63e7a5ebf6e9e85fc47a46a9f2b9bd57acea84b" exitCode=0 Nov 04 02:37:42.521721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.521708 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerDied","Data":"13040b301aaa20e78d73d484f63e7a5ebf6e9e85fc47a46a9f2b9bd57acea84b"} Nov 04 02:37:42.523889 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.523873 4784 generic.go:356] "Generic (PLEG): container finished" podID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerID="861d4db102ad8c27cca5ce85e588065dbf90b22203fe88f037051a53ced7ee5e" exitCode=0 Nov 04 02:37:42.524004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.523954 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerDied","Data":"861d4db102ad8c27cca5ce85e588065dbf90b22203fe88f037051a53ced7ee5e"} Nov 04 02:37:42.525980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.525952 4784 generic.go:356] "Generic (PLEG): container finished" podID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerID="0c5c9afbdfbff85da99a006cd584c682fb9d56d47327b9a3230cf6aa3acc1832" exitCode=0 Nov 04 02:37:42.526218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:42.526179 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" event={"ID":"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb","Type":"ContainerDied","Data":"0c5c9afbdfbff85da99a006cd584c682fb9d56d47327b9a3230cf6aa3acc1832"} Nov 04 02:37:43.535881 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.535845 4784 generic.go:356] "Generic (PLEG): container finished" podID="0f974998-5941-487b-bd70-f012ab01906f" containerID="564d6eec015a91519de396fdc455dc0a1169969f878c61e006e5133841aa18a2" exitCode=0 Nov 04 02:37:43.536345 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.535923 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" event={"ID":"0f974998-5941-487b-bd70-f012ab01906f","Type":"ContainerDied","Data":"564d6eec015a91519de396fdc455dc0a1169969f878c61e006e5133841aa18a2"} Nov 04 02:37:43.539354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.539329 4784 generic.go:356] "Generic (PLEG): container finished" podID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerID="822487e25dd32e5e48cb7519f1baa044ac8d122e25314c50b75a562150488eef" exitCode=0 Nov 04 02:37:43.539470 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.539453 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerDied","Data":"822487e25dd32e5e48cb7519f1baa044ac8d122e25314c50b75a562150488eef"} Nov 04 02:37:43.543165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.542640 4784 generic.go:356] "Generic (PLEG): container finished" podID="338ec91f-494a-405e-b297-a758f1ee79e3" containerID="ef303c4be555fd4b897dc380c2693e8d7662d41f3b3c4305df0d2e8806349265" exitCode=0 Nov 04 02:37:43.543165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:43.542732 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerDied","Data":"ef303c4be555fd4b897dc380c2693e8d7662d41f3b3c4305df0d2e8806349265"} Nov 04 02:37:49.609733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:49.609695 4784 generic.go:356] "Generic (PLEG): container finished" podID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerID="2d46766d61ba185f1c4e4e7ed4636eb1c3286f0770fc915840f0bc5f626e22f0" exitCode=0 Nov 04 02:37:49.610344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:49.609797 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerDied","Data":"2d46766d61ba185f1c4e4e7ed4636eb1c3286f0770fc915840f0bc5f626e22f0"} Nov 04 02:37:57.681416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:57.681372 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerStarted","Data":"3cbb45594de55dd7086a0157271e4d25c5e9595c7476e659927d74f9a4430347"} Nov 04 02:37:57.687560 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:57.686913 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerStarted","Data":"c96d30f388cb5a0d0dc1cc8677c93ffd621060f955f4988eed3a6bde63aae356"} Nov 04 02:37:57.808529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:57.806364 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" podStartSLOduration=2.506684231 podStartE2EDuration="34.806344938s" totalImagesPullingTime="32.299660707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:57.805817931 +0000 UTC m=+324.772318926" watchObservedRunningTime="2026-11-04 02:37:57.806344938 +0000 UTC m=+324.772845931" Nov 04 02:37:58.697121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.696458 4784 generic.go:356] "Generic (PLEG): container finished" podID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerID="c96d30f388cb5a0d0dc1cc8677c93ffd621060f955f4988eed3a6bde63aae356" exitCode=0 Nov 04 02:37:58.697121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.696985 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerDied","Data":"c96d30f388cb5a0d0dc1cc8677c93ffd621060f955f4988eed3a6bde63aae356"} Nov 04 02:37:58.700544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.700516 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerStarted","Data":"b5a15b7793ebeb269cc8ce79d629f76229b9a02aa35cea8081838cdebbc7f4a6"} Nov 04 02:37:58.703046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.703017 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerStarted","Data":"34553c80e2cdc4e16b89e01f832b37593bdbf0c61ef02c968fb55fd15a7815e7"} Nov 04 02:37:58.705717 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.705687 4784 generic.go:356] "Generic (PLEG): container finished" podID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerID="3cbb45594de55dd7086a0157271e4d25c5e9595c7476e659927d74f9a4430347" exitCode=0 Nov 04 02:37:58.705836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.705769 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerDied","Data":"3cbb45594de55dd7086a0157271e4d25c5e9595c7476e659927d74f9a4430347"} Nov 04 02:37:58.708114 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.708068 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerStarted","Data":"fe4dab0ce2c708f6c44baf595af8b13a15100101189e27bd679b9c8017447134"} Nov 04 02:37:58.710253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.710221 4784 generic.go:356] "Generic (PLEG): container finished" podID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerID="d5e552246127974519a78e75b8274c4989a7a37f571210ab88b2032b92b2d370" exitCode=0 Nov 04 02:37:58.710334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:58.710270 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" event={"ID":"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb","Type":"ContainerDied","Data":"d5e552246127974519a78e75b8274c4989a7a37f571210ab88b2032b92b2d370"} Nov 04 02:37:59.056772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.056665 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" podStartSLOduration=3.131106627 podStartE2EDuration="37.056652968s" totalImagesPullingTime="33.925546341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:59.055955221 +0000 UTC m=+326.022456214" watchObservedRunningTime="2026-11-04 02:37:59.056652968 +0000 UTC m=+326.023153961" Nov 04 02:37:59.296768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.296704 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" podStartSLOduration=3.100036781 podStartE2EDuration="34.296689651s" totalImagesPullingTime="31.19665287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:59.295708452 +0000 UTC m=+326.262209445" watchObservedRunningTime="2026-11-04 02:37:59.296689651 +0000 UTC m=+326.263190642" Nov 04 02:37:59.416736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.416672 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" podStartSLOduration=3.895846969 podStartE2EDuration="34.41665654s" totalImagesPullingTime="30.520809571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:37:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:37:59.415480872 +0000 UTC m=+326.381981864" watchObservedRunningTime="2026-11-04 02:37:59.41665654 +0000 UTC m=+326.383157531" Nov 04 02:37:59.716755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.716668 4784 generic.go:356] "Generic (PLEG): container finished" podID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerID="34553c80e2cdc4e16b89e01f832b37593bdbf0c61ef02c968fb55fd15a7815e7" exitCode=0 Nov 04 02:37:59.717238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.716748 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerDied","Data":"34553c80e2cdc4e16b89e01f832b37593bdbf0c61ef02c968fb55fd15a7815e7"} Nov 04 02:37:59.718769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.718745 4784 generic.go:356] "Generic (PLEG): container finished" podID="338ec91f-494a-405e-b297-a758f1ee79e3" containerID="fe4dab0ce2c708f6c44baf595af8b13a15100101189e27bd679b9c8017447134" exitCode=0 Nov 04 02:37:59.718895 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.718824 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerDied","Data":"fe4dab0ce2c708f6c44baf595af8b13a15100101189e27bd679b9c8017447134"} Nov 04 02:37:59.720801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.720780 4784 generic.go:356] "Generic (PLEG): container finished" podID="0f974998-5941-487b-bd70-f012ab01906f" containerID="1013fed8a42667dadca70f094d91c8d2ee05e978c231c028c49eb9d54ba36669" exitCode=0 Nov 04 02:37:59.720903 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.720846 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" event={"ID":"0f974998-5941-487b-bd70-f012ab01906f","Type":"ContainerDied","Data":"1013fed8a42667dadca70f094d91c8d2ee05e978c231c028c49eb9d54ba36669"} Nov 04 02:37:59.722721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.722701 4784 generic.go:356] "Generic (PLEG): container finished" podID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerID="b5a15b7793ebeb269cc8ce79d629f76229b9a02aa35cea8081838cdebbc7f4a6" exitCode=0 Nov 04 02:37:59.722810 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.722793 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerDied","Data":"b5a15b7793ebeb269cc8ce79d629f76229b9a02aa35cea8081838cdebbc7f4a6"} Nov 04 02:37:59.862688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.862660 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:37:59.888835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.888811 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:37:59.896692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.896670 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:37:59.929197 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929167 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\") pod \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " Nov 04 02:37:59.929353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929227 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\") pod \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " Nov 04 02:37:59.929353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929269 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\" (UniqueName: \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\") pod \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " Nov 04 02:37:59.929353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929295 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\") pod \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " Nov 04 02:37:59.929353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929321 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\") pod \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " Nov 04 02:37:59.929353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929340 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\") pod \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " Nov 04 02:37:59.929626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929373 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\" (UniqueName: \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\") pod \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\" (UID: \"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb\") " Nov 04 02:37:59.929626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929411 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\" (UniqueName: \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\") pod \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\" (UID: \"48cbd183-02fa-4b53-b13b-7f8eaf2095af\") " Nov 04 02:37:59.929626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.929452 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\") pod \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\" (UID: \"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419\") " Nov 04 02:37:59.930236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.930206 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle" pod "48cbd183-02fa-4b53-b13b-7f8eaf2095af" (UID: "48cbd183-02fa-4b53-b13b-7f8eaf2095af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:37:59.930320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.930283 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle" pod "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" (UID: "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:37:59.930668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.930647 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle" pod "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" (UID: "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:37:59.932628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.932588 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc" pod "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" (UID: "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419"). InnerVolumeSpecName "kube-api-access-l6hrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:37:59.932628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.932580 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2" pod "48cbd183-02fa-4b53-b13b-7f8eaf2095af" (UID: "48cbd183-02fa-4b53-b13b-7f8eaf2095af"). InnerVolumeSpecName "kube-api-access-486w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:37:59.932912 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.932893 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg" pod "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" (UID: "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb"). InnerVolumeSpecName "kube-api-access-4tdvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:37:59.935541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.935518 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util" pod "48cbd183-02fa-4b53-b13b-7f8eaf2095af" (UID: "48cbd183-02fa-4b53-b13b-7f8eaf2095af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:37:59.936242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.936218 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util" pod "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" (UID: "7c38ea37-f28b-4f84-b86f-cd3f5f0b1419"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:37:59.936295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:37:59.936218 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util" pod "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" (UID: "2ef0c230-1c45-47ed-89a3-bfc87b0b57fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030320 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030346 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030355 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030364 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hrc\" (UniqueName: \"kubernetes.io/projected/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-kube-api-access-l6hrc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030372 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48cbd183-02fa-4b53-b13b-7f8eaf2095af-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030380 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030388 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c38ea37-f28b-4f84-b86f-cd3f5f0b1419-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030396 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tdvg\" (UniqueName: \"kubernetes.io/projected/2ef0c230-1c45-47ed-89a3-bfc87b0b57fb-kube-api-access-4tdvg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.030401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.030404 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486w2\" (UniqueName: \"kubernetes.io/projected/48cbd183-02fa-4b53-b13b-7f8eaf2095af-kube-api-access-486w2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:00.728698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.728665 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" Nov 04 02:38:00.729176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.728661 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwvsw" event={"ID":"48cbd183-02fa-4b53-b13b-7f8eaf2095af","Type":"ContainerDied","Data":"df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388"} Nov 04 02:38:00.729176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.728827 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df0de576f85d1e2f456203c311b62ea4a806a344130263f3d7676c114adc1388" Nov 04 02:38:00.730831 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.730800 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" Nov 04 02:38:00.730933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.730819 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tjknh" event={"ID":"2ef0c230-1c45-47ed-89a3-bfc87b0b57fb","Type":"ContainerDied","Data":"abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0"} Nov 04 02:38:00.730933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.730866 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abd8d2ea70c5fbef1f6496adc2117602395f7176d2349f1f4b17ae49e258e8c0" Nov 04 02:38:00.732949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.732931 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" Nov 04 02:38:00.733050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.732969 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ljt7m" event={"ID":"7c38ea37-f28b-4f84-b86f-cd3f5f0b1419","Type":"ContainerDied","Data":"93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42"} Nov 04 02:38:00.733050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.733015 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c019ac651a941965a2785ea2cf3c1392aa3147d644d38a6d64f6197e12ec42" Nov 04 02:38:00.880750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.880725 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:38:00.913660 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.913632 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:38:00.916659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.916639 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:38:00.934849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.934829 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\" (UniqueName: \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\") pod \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " Nov 04 02:38:00.934922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.934882 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\" (UniqueName: \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\") pod \"338ec91f-494a-405e-b297-a758f1ee79e3\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " Nov 04 02:38:00.934922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.934915 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\" (UniqueName: \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\") pod \"0f974998-5941-487b-bd70-f012ab01906f\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " Nov 04 02:38:00.935054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935034 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\") pod \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " Nov 04 02:38:00.935119 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935080 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\") pod \"0f974998-5941-487b-bd70-f012ab01906f\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " Nov 04 02:38:00.935172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935115 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\") pod \"0f974998-5941-487b-bd70-f012ab01906f\" (UID: \"0f974998-5941-487b-bd70-f012ab01906f\") " Nov 04 02:38:00.935172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935157 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\") pod \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\" (UID: \"b97499d7-b502-4fe8-ac1f-65ec37884e3c\") " Nov 04 02:38:00.935322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935242 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\") pod \"338ec91f-494a-405e-b297-a758f1ee79e3\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " Nov 04 02:38:00.935322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.935275 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\") pod \"338ec91f-494a-405e-b297-a758f1ee79e3\" (UID: \"338ec91f-494a-405e-b297-a758f1ee79e3\") " Nov 04 02:38:00.936422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.936385 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle" pod "b97499d7-b502-4fe8-ac1f-65ec37884e3c" (UID: "b97499d7-b502-4fe8-ac1f-65ec37884e3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.937120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.937089 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle" pod "0f974998-5941-487b-bd70-f012ab01906f" (UID: "0f974998-5941-487b-bd70-f012ab01906f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.937236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.937170 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42" pod "b97499d7-b502-4fe8-ac1f-65ec37884e3c" (UID: "b97499d7-b502-4fe8-ac1f-65ec37884e3c"). InnerVolumeSpecName "kube-api-access-29s42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:38:00.937412 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.937393 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz" pod "338ec91f-494a-405e-b297-a758f1ee79e3" (UID: "338ec91f-494a-405e-b297-a758f1ee79e3"). InnerVolumeSpecName "kube-api-access-nc8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:38:00.937569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.937552 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5" pod "0f974998-5941-487b-bd70-f012ab01906f" (UID: "0f974998-5941-487b-bd70-f012ab01906f"). InnerVolumeSpecName "kube-api-access-zltx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:38:00.937657 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.937638 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle" pod "338ec91f-494a-405e-b297-a758f1ee79e3" (UID: "338ec91f-494a-405e-b297-a758f1ee79e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.942642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.942620 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util" pod "0f974998-5941-487b-bd70-f012ab01906f" (UID: "0f974998-5941-487b-bd70-f012ab01906f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.943467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.943445 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util" pod "b97499d7-b502-4fe8-ac1f-65ec37884e3c" (UID: "b97499d7-b502-4fe8-ac1f-65ec37884e3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.943594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.943574 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util" pod "338ec91f-494a-405e-b297-a758f1ee79e3" (UID: "338ec91f-494a-405e-b297-a758f1ee79e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:00.945706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:00.945686 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:38:01.035766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.035696 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\") pod \"941dd8f5-6740-4881-9bde-e55346a1ad23\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " Nov 04 02:38:01.035766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.035731 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\") pod \"941dd8f5-6740-4881-9bde-e55346a1ad23\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " Nov 04 02:38:01.035940 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.035773 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\" (UniqueName: \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\") pod \"941dd8f5-6740-4881-9bde-e55346a1ad23\" (UID: \"941dd8f5-6740-4881-9bde-e55346a1ad23\") " Nov 04 02:38:01.036040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036021 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc8vz\" (UniqueName: \"kubernetes.io/projected/338ec91f-494a-405e-b297-a758f1ee79e3-kube-api-access-nc8vz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036048 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zltx5\" (UniqueName: \"kubernetes.io/projected/0f974998-5941-487b-bd70-f012ab01906f-kube-api-access-zltx5\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036063 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036077 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036090 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f974998-5941-487b-bd70-f012ab01906f-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036104 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97499d7-b502-4fe8-ac1f-65ec37884e3c-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036117 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036129 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338ec91f-494a-405e-b297-a758f1ee79e3-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036126 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle" pod "941dd8f5-6740-4881-9bde-e55346a1ad23" (UID: "941dd8f5-6740-4881-9bde-e55346a1ad23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:01.036203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.036141 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29s42\" (UniqueName: \"kubernetes.io/projected/b97499d7-b502-4fe8-ac1f-65ec37884e3c-kube-api-access-29s42\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.038042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.038014 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s" pod "941dd8f5-6740-4881-9bde-e55346a1ad23" (UID: "941dd8f5-6740-4881-9bde-e55346a1ad23"). InnerVolumeSpecName "kube-api-access-7fc8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:38:01.041428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.041406 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util" pod "941dd8f5-6740-4881-9bde-e55346a1ad23" (UID: "941dd8f5-6740-4881-9bde-e55346a1ad23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:38:01.136785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.136748 4784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-util\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.136785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.136778 4784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941dd8f5-6740-4881-9bde-e55346a1ad23-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.136785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.136789 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fc8s\" (UniqueName: \"kubernetes.io/projected/941dd8f5-6740-4881-9bde-e55346a1ad23-kube-api-access-7fc8s\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:38:01.739593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.739565 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" Nov 04 02:38:01.740003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.739560 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abtpd9" event={"ID":"0f974998-5941-487b-bd70-f012ab01906f","Type":"ContainerDied","Data":"a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a"} Nov 04 02:38:01.740003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.739700 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3663820ff922ff07504f9310cc3c131ab34537e4a46535c3e472b2a6dad147a" Nov 04 02:38:01.741703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.741675 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" event={"ID":"b97499d7-b502-4fe8-ac1f-65ec37884e3c","Type":"ContainerDied","Data":"17c7d80eb87ca509e9887b3e62bbf687a7833527b7811b00e9c380f099449360"} Nov 04 02:38:01.741861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.741712 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sjmqf" Nov 04 02:38:01.741861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.741730 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c7d80eb87ca509e9887b3e62bbf687a7833527b7811b00e9c380f099449360" Nov 04 02:38:01.744235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.744214 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" event={"ID":"941dd8f5-6740-4881-9bde-e55346a1ad23","Type":"ContainerDied","Data":"dd1924d7e7359215efc88d1002a51f0283c885593ef15e864dbe90f15cd7493d"} Nov 04 02:38:01.744366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.744252 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1924d7e7359215efc88d1002a51f0283c885593ef15e864dbe90f15cd7493d" Nov 04 02:38:01.744366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.744220 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444j4xf" Nov 04 02:38:01.746179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.746154 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" Nov 04 02:38:01.746179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.746146 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7pct4" event={"ID":"338ec91f-494a-405e-b297-a758f1ee79e3","Type":"ContainerDied","Data":"b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248"} Nov 04 02:38:01.746350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:01.746228 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b340fee14e9cc4dfb47fe7fc9fa23b15983e695d30bcc235bea5a541d2742248" Nov 04 02:38:06.398019 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:38:06.397982 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:38:12.311914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.311874 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm"] Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312346 4784 cpu_manager.go:414] "containerMap: removing container" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312365 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312381 4784 cpu_manager.go:414] "containerMap: removing container" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312388 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312399 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312406 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312432 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312439 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312449 4784 cpu_manager.go:414] "containerMap: removing container" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312455 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312468 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312474 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312484 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312492 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312503 4784 cpu_manager.go:414] "containerMap: removing container" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312510 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="pull" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312519 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312527 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="extract" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312545 4784 cpu_manager.go:414] "containerMap: removing container" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312553 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312562 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312569 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312577 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312584 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312601 4784 cpu_manager.go:414] "containerMap: removing container" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312607 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="util" Nov 04 02:38:12.312601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312615 4784 cpu_manager.go:414] "containerMap: removing container" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="util" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312621 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="util" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312633 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="util" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312639 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="util" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312649 4784 cpu_manager.go:414] "containerMap: removing container" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="pull" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312656 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="pull" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312665 4784 cpu_manager.go:414] "containerMap: removing container" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312672 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312680 4784 cpu_manager.go:414] "containerMap: removing container" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312686 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312698 4784 cpu_manager.go:414] "containerMap: removing container" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="pull" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312704 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="pull" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312713 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312720 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312729 4784 cpu_manager.go:414] "containerMap: removing container" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312735 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312813 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="48cbd183-02fa-4b53-b13b-7f8eaf2095af" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312826 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f974998-5941-487b-bd70-f012ab01906f" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312835 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97499d7-b502-4fe8-ac1f-65ec37884e3c" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312845 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="941dd8f5-6740-4881-9bde-e55346a1ad23" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312856 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="338ec91f-494a-405e-b297-a758f1ee79e3" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312865 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c38ea37-f28b-4f84-b86f-cd3f5f0b1419" containerName="extract" Nov 04 02:38:12.313837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.312872 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef0c230-1c45-47ed-89a3-bfc87b0b57fb" containerName="extract" Nov 04 02:38:12.316424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.316401 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.370655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370623 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 02:38:12.370857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370737 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 02:38:12.370857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370762 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 02:38:12.370857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370799 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 02:38:12.370997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370885 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8qrsw\"" Nov 04 02:38:12.370997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.370919 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 02:38:12.411836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.411801 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.411836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.411838 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52kt\" (UniqueName: \"kubernetes.io/projected/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-kube-api-access-w52kt\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.412031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.411871 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.412031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.411957 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.412031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.411980 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.412031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.412008 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.432125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.432095 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm"] Nov 04 02:38:12.512610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512575 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.512610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512612 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52kt\" (UniqueName: \"kubernetes.io/projected/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-kube-api-access-w52kt\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.512878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512633 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.512878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512695 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.512878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512731 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.512878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.512778 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.513244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.513210 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.515036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.515004 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.515250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.515234 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.515351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.515286 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.515351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.515340 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.729635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.729598 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52kt\" (UniqueName: \"kubernetes.io/projected/dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3-kube-api-access-w52kt\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm\" (UID: \"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:12.931373 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:12.931339 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:13.528622 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:13.528594 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7ba68d_6ca3_4d54_9dd4_d84a26465eb3.slice/crio-3d5511d4311d4533d73a7476a79d830334137edcdd4297111e90e85ed73f0f01 WatchSource:0}: Error finding container 3d5511d4311d4533d73a7476a79d830334137edcdd4297111e90e85ed73f0f01: Status 404 returned error can't find the container with id 3d5511d4311d4533d73a7476a79d830334137edcdd4297111e90e85ed73f0f01 Nov 04 02:38:13.577948 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:13.577909 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm"] Nov 04 02:38:13.799450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:13.799364 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" event={"ID":"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3","Type":"ContainerStarted","Data":"3d5511d4311d4533d73a7476a79d830334137edcdd4297111e90e85ed73f0f01"} Nov 04 02:38:17.819439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:17.819400 4784 generic.go:356] "Generic (PLEG): container finished" podID="dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3" containerID="5e7086868793b130e416f1b3084a89e0379dbe91f2bbd10a67659d8db3d0643a" exitCode=0 Nov 04 02:38:17.819934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:17.819490 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" event={"ID":"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3","Type":"ContainerDied","Data":"5e7086868793b130e416f1b3084a89e0379dbe91f2bbd10a67659d8db3d0643a"} Nov 04 02:38:18.609654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.609615 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv"] Nov 04 02:38:18.613163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.613146 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.657049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.657001 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf29d\" (UniqueName: \"kubernetes.io/projected/b66c9d7b-4f94-4d19-a25d-e1f22a477595-kube-api-access-nf29d\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.657240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.657122 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b66c9d7b-4f94-4d19-a25d-e1f22a477595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.667113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.667083 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xjtxc\"" Nov 04 02:38:18.667242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.667125 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:38:18.667242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.667147 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 04 02:38:18.678728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.678693 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv"] Nov 04 02:38:18.757958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.757924 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf29d\" (UniqueName: \"kubernetes.io/projected/b66c9d7b-4f94-4d19-a25d-e1f22a477595-kube-api-access-nf29d\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.758136 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.757978 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b66c9d7b-4f94-4d19-a25d-e1f22a477595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.758343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.758325 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b66c9d7b-4f94-4d19-a25d-e1f22a477595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.820900 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.820869 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf29d\" (UniqueName: \"kubernetes.io/projected/b66c9d7b-4f94-4d19-a25d-e1f22a477595-kube-api-access-nf29d\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv\" (UID: \"b66c9d7b-4f94-4d19-a25d-e1f22a477595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:18.924323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:18.924289 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" Nov 04 02:38:19.402170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:19.402145 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv"] Nov 04 02:38:19.403908 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:19.403873 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66c9d7b_4f94_4d19_a25d_e1f22a477595.slice/crio-3596257fa647e1ea1dfbfd4b3cbcfb77199fdf70c93c3b3e7233ab3ef962d5ad WatchSource:0}: Error finding container 3596257fa647e1ea1dfbfd4b3cbcfb77199fdf70c93c3b3e7233ab3ef962d5ad: Status 404 returned error can't find the container with id 3596257fa647e1ea1dfbfd4b3cbcfb77199fdf70c93c3b3e7233ab3ef962d5ad Nov 04 02:38:19.830206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:19.830151 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" event={"ID":"b66c9d7b-4f94-4d19-a25d-e1f22a477595","Type":"ContainerStarted","Data":"3596257fa647e1ea1dfbfd4b3cbcfb77199fdf70c93c3b3e7233ab3ef962d5ad"} Nov 04 02:38:19.832269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:19.832244 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" event={"ID":"dd7ba68d-6ca3-4d54-9dd4-d84a26465eb3","Type":"ContainerStarted","Data":"494c797e959262066ba9b3b1743c0cc08df215f1420851ccc7d738d34bb8e325"} Nov 04 02:38:19.832405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:19.832387 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:20.082828 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:20.082704 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" podStartSLOduration=3.359598463 podStartE2EDuration="8.082684798s" totalImagesPullingTime="4.255178821s" totalInitContainerRuntime="467.907514ms" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:20.030714005 +0000 UTC m=+346.997214999" watchObservedRunningTime="2026-11-04 02:38:20.082684798 +0000 UTC m=+347.049185789" Nov 04 02:38:23.852481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:23.852445 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" event={"ID":"b66c9d7b-4f94-4d19-a25d-e1f22a477595","Type":"ContainerStarted","Data":"c083092dd614ca320d6ce0b3968ffadf826d8df343eb805a3a5c525312766963"} Nov 04 02:38:24.080454 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:24.080377 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8p5vv" podStartSLOduration=2.115182058 podStartE2EDuration="6.080362814s" totalImagesPullingTime="3.965180756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:24.028079257 +0000 UTC m=+350.994580250" watchObservedRunningTime="2026-11-04 02:38:24.080362814 +0000 UTC m=+351.046863805" Nov 04 02:38:28.301235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.301184 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j"] Nov 04 02:38:28.304467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.304445 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:28.334453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.334421 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7qr\" (UniqueName: \"kubernetes.io/projected/b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a-kube-api-access-gt7qr\") pod \"dns-operator-controller-manager-8557cbdb84-9vc9j\" (UID: \"b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:28.362443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.362413 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 02:38:28.362443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.362418 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 02:38:28.380267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.380234 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 02:38:28.380441 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.380343 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fchrb\"" Nov 04 02:38:28.412120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.412089 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j"] Nov 04 02:38:28.435064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.435024 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7qr\" (UniqueName: \"kubernetes.io/projected/b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a-kube-api-access-gt7qr\") pod \"dns-operator-controller-manager-8557cbdb84-9vc9j\" (UID: \"b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:28.500913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.500876 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7qr\" (UniqueName: \"kubernetes.io/projected/b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a-kube-api-access-gt7qr\") pod \"dns-operator-controller-manager-8557cbdb84-9vc9j\" (UID: \"b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:28.616568 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:28.616467 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:29.008702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:29.008651 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j"] Nov 04 02:38:29.011097 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:29.011066 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280d45c_ddd2_4c1d_b0f0_d2aff81fba7a.slice/crio-35833d4cafcc8b68d625f44f63a48aec012ba1761c1dafbf82b8744095fa8510 WatchSource:0}: Error finding container 35833d4cafcc8b68d625f44f63a48aec012ba1761c1dafbf82b8744095fa8510: Status 404 returned error can't find the container with id 35833d4cafcc8b68d625f44f63a48aec012ba1761c1dafbf82b8744095fa8510 Nov 04 02:38:29.886427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:29.886384 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" event={"ID":"b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a","Type":"ContainerStarted","Data":"35833d4cafcc8b68d625f44f63a48aec012ba1761c1dafbf82b8744095fa8510"} Nov 04 02:38:30.840171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:30.840142 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-m4gpm" Nov 04 02:38:33.202148 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:33.202118 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 02:38:33.906679 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:33.906640 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" event={"ID":"b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a","Type":"ContainerStarted","Data":"bef4f9a8b981887b88d07576707e8506560a9055bb67816651e1bdecebfe24dc"} Nov 04 02:38:33.906895 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:33.906768 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:34.035322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:34.035240 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" podStartSLOduration=1.89982399 podStartE2EDuration="6.035226032s" totalImagesPullingTime="4.135402042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:34.032983122 +0000 UTC m=+360.999484112" watchObservedRunningTime="2026-11-04 02:38:34.035226032 +0000 UTC m=+361.001727022" Nov 04 02:38:36.037713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.037679 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs"] Nov 04 02:38:36.041266 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.041244 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.105224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.105165 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.105406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.105337 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9vr\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-kube-api-access-qq9vr\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.137292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.137255 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 02:38:36.137292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.137277 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hnwd6\"" Nov 04 02:38:36.137292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.137288 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 02:38:36.139004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.138986 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs"] Nov 04 02:38:36.206250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.206213 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9vr\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-kube-api-access-qq9vr\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.206418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.206282 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.268276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.268241 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.268443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.268362 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9vr\" (UniqueName: \"kubernetes.io/projected/7223d9eb-c25d-41b2-b91b-856813c12b60-kube-api-access-qq9vr\") pod \"cert-manager-cainjector-5c57bbd4fc-fsdjs\" (UID: \"7223d9eb-c25d-41b2-b91b-856813c12b60\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.364255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.364149 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" Nov 04 02:38:36.400877 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:38:36.400841 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:38:36.531441 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.531407 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jmrrs"] Nov 04 02:38:36.536090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.536061 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.590087 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.590057 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t5g54\"" Nov 04 02:38:36.592094 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.592066 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jmrrs"] Nov 04 02:38:36.610548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.610515 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-bound-sa-token\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.610548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.610548 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnsjt\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-kube-api-access-mnsjt\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.711233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.711180 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-bound-sa-token\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.711233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.711232 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnsjt\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-kube-api-access-mnsjt\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.769549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.769527 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs"] Nov 04 02:38:36.771063 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:36.771032 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7223d9eb_c25d_41b2_b91b_856813c12b60.slice/crio-986080dd7fcf7872ffd79ba096d92c708eb2239c3943646006807e8defa5097c WatchSource:0}: Error finding container 986080dd7fcf7872ffd79ba096d92c708eb2239c3943646006807e8defa5097c: Status 404 returned error can't find the container with id 986080dd7fcf7872ffd79ba096d92c708eb2239c3943646006807e8defa5097c Nov 04 02:38:36.773756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.773732 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:38:36.777569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.777551 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.786528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.786506 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-bound-sa-token\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.786715 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.786694 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnsjt\" (UniqueName: \"kubernetes.io/projected/a4893f08-abfd-4fb3-9681-660b7c631c6e-kube-api-access-mnsjt\") pod \"cert-manager-597b95cb84-jmrrs\" (UID: \"a4893f08-abfd-4fb3-9681-660b7c631c6e\") " pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.812288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.812249 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghss\" (UniqueName: \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.812466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.812365 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.833921 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.833891 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pbzpn\"" Nov 04 02:38:36.849063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.849030 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jmrrs" Nov 04 02:38:36.876695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.876660 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:38:36.914426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.914394 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.914572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.914513 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.914628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.914616 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghss\" (UniqueName: \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:36.927614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.927578 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" event={"ID":"7223d9eb-c25d-41b2-b91b-856813c12b60","Type":"ContainerStarted","Data":"986080dd7fcf7872ffd79ba096d92c708eb2239c3943646006807e8defa5097c"} Nov 04 02:38:36.980063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:36.979987 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghss\" (UniqueName: \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7qfrw\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:37.090963 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:37.090927 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:37.225777 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:37.225747 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jmrrs"] Nov 04 02:38:37.226687 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:37.226656 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4893f08_abfd_4fb3_9681_660b7c631c6e.slice/crio-78e3ccfd51db067dc23633314ec08c6b96919f9012aab5bc102a798aabe4751d WatchSource:0}: Error finding container 78e3ccfd51db067dc23633314ec08c6b96919f9012aab5bc102a798aabe4751d: Status 404 returned error can't find the container with id 78e3ccfd51db067dc23633314ec08c6b96919f9012aab5bc102a798aabe4751d Nov 04 02:38:37.476643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:37.476523 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:38:37.478717 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:37.478683 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e223a20_f5c8_430d_9a54_76f78577a543.slice/crio-e3dbc041f1a9e21692f2b9a1171f8a8b4ca302223e30dfc4565a897f59c6d556 WatchSource:0}: Error finding container e3dbc041f1a9e21692f2b9a1171f8a8b4ca302223e30dfc4565a897f59c6d556: Status 404 returned error can't find the container with id e3dbc041f1a9e21692f2b9a1171f8a8b4ca302223e30dfc4565a897f59c6d556 Nov 04 02:38:37.934172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:37.934135 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" event={"ID":"2e223a20-f5c8-430d-9a54-76f78577a543","Type":"ContainerStarted","Data":"e3dbc041f1a9e21692f2b9a1171f8a8b4ca302223e30dfc4565a897f59c6d556"} Nov 04 02:38:37.935334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:37.935290 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jmrrs" event={"ID":"a4893f08-abfd-4fb3-9681-660b7c631c6e","Type":"ContainerStarted","Data":"78e3ccfd51db067dc23633314ec08c6b96919f9012aab5bc102a798aabe4751d"} Nov 04 02:38:40.955958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:40.955911 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" event={"ID":"7223d9eb-c25d-41b2-b91b-856813c12b60","Type":"ContainerStarted","Data":"3b2ca50e3132f3ca5878f001b08ed24a5cde62692fbad41ba8f71f0e7386469a"} Nov 04 02:38:41.075849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:41.075785 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fsdjs" podStartSLOduration=2.264540624 podStartE2EDuration="6.075766417s" totalImagesPullingTime="3.811225793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:41.075392435 +0000 UTC m=+368.041893428" watchObservedRunningTime="2026-11-04 02:38:41.075766417 +0000 UTC m=+368.042267408" Nov 04 02:38:41.965426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:41.965293 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jmrrs" event={"ID":"a4893f08-abfd-4fb3-9681-660b7c631c6e","Type":"ContainerStarted","Data":"8ec49b5658b1e5f15c52354c326ca3611d7f8cb2d6782ec533cea27d17c8d3f4"} Nov 04 02:38:42.093960 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:42.093905 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jmrrs" podStartSLOduration=2.427595526 podStartE2EDuration="6.093885327s" totalImagesPullingTime="3.666289801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:42.093246269 +0000 UTC m=+369.059747261" watchObservedRunningTime="2026-11-04 02:38:42.093885327 +0000 UTC m=+369.060386319" Nov 04 02:38:42.971710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:42.971668 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" event={"ID":"2e223a20-f5c8-430d-9a54-76f78577a543","Type":"ContainerStarted","Data":"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca"} Nov 04 02:38:42.972262 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:42.971868 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:43.197642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.197587 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" podStartSLOduration=2.041908802 podStartE2EDuration="7.197573522s" totalImagesPullingTime="5.15566472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:43.146347721 +0000 UTC m=+370.112848711" watchObservedRunningTime="2026-11-04 02:38:43.197573522 +0000 UTC m=+370.164074513" Nov 04 02:38:43.924655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.924619 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj"] Nov 04 02:38:43.928087 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.928069 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.980598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980563 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-kube-api-access-2crt4\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.980598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980600 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980632 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f832f885-5e72-4892-8425-95fd377a536a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980764 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980804 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980825 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980839 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980859 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.981089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.980945 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f832f885-5e72-4892-8425-95fd377a536a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:43.991034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:43.991004 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj"] Nov 04 02:38:44.054867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.054789 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q5vcl\"" Nov 04 02:38:44.081875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-kube-api-access-2crt4\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.081875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081876 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081929 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f832f885-5e72-4892-8425-95fd377a536a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081966 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081985 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.081999 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082013 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082033 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082137 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082061 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f832f885-5e72-4892-8425-95fd377a536a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082632 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082598 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082665 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082730 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.082981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.082919 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f832f885-5e72-4892-8425-95fd377a536a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.083105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.083075 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.084680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.084650 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f832f885-5e72-4892-8425-95fd377a536a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.084757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.084716 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f832f885-5e72-4892-8425-95fd377a536a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.147822 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.147787 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-kube-api-access-2crt4\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.148616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.148598 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f832f885-5e72-4892-8425-95fd377a536a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b4848996xtj\" (UID: \"f832f885-5e72-4892-8425-95fd377a536a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.240507 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.240418 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:44.681049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.681023 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj"] Nov 04 02:38:44.682207 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:44.682157 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf832f885_5e72_4892_8425_95fd377a536a.slice/crio-4b677dfe7d43729be355d92db4e5aea960e583dc369954961c920b1f296363b5 WatchSource:0}: Error finding container 4b677dfe7d43729be355d92db4e5aea960e583dc369954961c920b1f296363b5: Status 404 returned error can't find the container with id 4b677dfe7d43729be355d92db4e5aea960e583dc369954961c920b1f296363b5 Nov 04 02:38:44.684416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.684371 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:38:44.684495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.684448 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:38:44.684495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.684476 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:38:44.914201 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.914151 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9vc9j" Nov 04 02:38:44.983737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.983640 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" event={"ID":"f832f885-5e72-4892-8425-95fd377a536a","Type":"ContainerStarted","Data":"ac66309998fb59de1839751bd865fd439d9ec8fa2454442dde7188025eeb1906"} Nov 04 02:38:44.983737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:44.983701 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" event={"ID":"f832f885-5e72-4892-8425-95fd377a536a","Type":"ContainerStarted","Data":"4b677dfe7d43729be355d92db4e5aea960e583dc369954961c920b1f296363b5"} Nov 04 02:38:45.165746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.165696 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" podStartSLOduration=2.165683682 podStartE2EDuration="2.165683682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:45.164911014 +0000 UTC m=+372.131412042" watchObservedRunningTime="2026-11-04 02:38:45.165683682 +0000 UTC m=+372.132184672" Nov 04 02:38:45.202090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.202052 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs"] Nov 04 02:38:45.205535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.205519 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:45.240646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.240555 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:45.245690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.245659 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:45.265140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.265108 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bplrg\"" Nov 04 02:38:45.289931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.289669 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs"] Nov 04 02:38:45.293635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.293604 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2dk\" (UniqueName: \"kubernetes.io/projected/792ef3b5-5678-4de5-9764-86a3b17138c8-kube-api-access-hx2dk\") pod \"limitador-operator-controller-manager-5778bb4449-68mzs\" (UID: \"792ef3b5-5678-4de5-9764-86a3b17138c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:45.394386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.394350 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2dk\" (UniqueName: \"kubernetes.io/projected/792ef3b5-5678-4de5-9764-86a3b17138c8-kube-api-access-hx2dk\") pod \"limitador-operator-controller-manager-5778bb4449-68mzs\" (UID: \"792ef3b5-5678-4de5-9764-86a3b17138c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:45.463528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.463498 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2dk\" (UniqueName: \"kubernetes.io/projected/792ef3b5-5678-4de5-9764-86a3b17138c8-kube-api-access-hx2dk\") pod \"limitador-operator-controller-manager-5778bb4449-68mzs\" (UID: \"792ef3b5-5678-4de5-9764-86a3b17138c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:45.519031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.518935 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:45.894980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.894954 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs"] Nov 04 02:38:45.896735 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:45.896706 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792ef3b5_5678_4de5_9764_86a3b17138c8.slice/crio-2d85e5e16dc086f932ca94497877c6f3180b422ff496538d729664051e2e8bd1 WatchSource:0}: Error finding container 2d85e5e16dc086f932ca94497877c6f3180b422ff496538d729664051e2e8bd1: Status 404 returned error can't find the container with id 2d85e5e16dc086f932ca94497877c6f3180b422ff496538d729664051e2e8bd1 Nov 04 02:38:45.989790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.989752 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" event={"ID":"792ef3b5-5678-4de5-9764-86a3b17138c8","Type":"ContainerStarted","Data":"2d85e5e16dc086f932ca94497877c6f3180b422ff496538d729664051e2e8bd1"} Nov 04 02:38:45.990234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.989963 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:45.991005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:45.990991 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b4848996xtj" Nov 04 02:38:50.012967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:50.012933 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" event={"ID":"792ef3b5-5678-4de5-9764-86a3b17138c8","Type":"ContainerStarted","Data":"7603d6570b015cd00cf1148c21662b29415693c4a0f50666d9c77a6c7421cf13"} Nov 04 02:38:50.013450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:50.013065 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:38:50.132369 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:50.132312 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" podStartSLOduration=2.049489117 podStartE2EDuration="5.132296492s" totalImagesPullingTime="3.082807375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:50.131131157 +0000 UTC m=+377.097632148" watchObservedRunningTime="2026-11-04 02:38:50.132296492 +0000 UTC m=+377.098797484" Nov 04 02:38:53.979813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:53.979774 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:38:55.851552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.851516 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j"] Nov 04 02:38:55.856759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.856741 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:55.914203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.914153 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-ggpwx\"" Nov 04 02:38:55.967056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.967023 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j"] Nov 04 02:38:55.969076 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.969055 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 02:38:55.969262 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.969246 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 02:38:55.990707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.990670 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:55.990850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.990766 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e830410b-0936-4d99-9932-ffa515c1760a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:55.990850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.990827 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx44z\" (UniqueName: \"kubernetes.io/projected/e830410b-0936-4d99-9932-ffa515c1760a-kube-api-access-nx44z\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:55.990974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:55.990899 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.071700 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.071661 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 02:38:56.092180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092144 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.092364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092228 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.092364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092269 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e830410b-0936-4d99-9932-ffa515c1760a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.092364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092313 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx44z\" (UniqueName: \"kubernetes.io/projected/e830410b-0936-4d99-9932-ffa515c1760a-kube-api-access-nx44z\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.092654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092630 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.092689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.092670 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e830410b-0936-4d99-9932-ffa515c1760a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.093060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.093041 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e830410b-0936-4d99-9932-ffa515c1760a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.154690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.154620 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx44z\" (UniqueName: \"kubernetes.io/projected/e830410b-0936-4d99-9932-ffa515c1760a-kube-api-access-nx44z\") pod \"kserve-module-controller-manager-7566cf5fc8-k7c2j\" (UID: \"e830410b-0936-4d99-9932-ffa515c1760a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.170130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.170087 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:38:56.600865 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:56.600841 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j"] Nov 04 02:38:56.602532 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:38:56.602495 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode830410b_0936_4d99_9932_ffa515c1760a.slice/crio-7a74ff1e6daa2552bd96e8d376a5af64a9de79a64596f87ebe12da193f5c4cba WatchSource:0}: Error finding container 7a74ff1e6daa2552bd96e8d376a5af64a9de79a64596f87ebe12da193f5c4cba: Status 404 returned error can't find the container with id 7a74ff1e6daa2552bd96e8d376a5af64a9de79a64596f87ebe12da193f5c4cba Nov 04 02:38:57.048923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:38:57.048884 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" event={"ID":"e830410b-0936-4d99-9932-ffa515c1760a","Type":"ContainerStarted","Data":"7a74ff1e6daa2552bd96e8d376a5af64a9de79a64596f87ebe12da193f5c4cba"} Nov 04 02:39:01.020842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.020810 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-68mzs" Nov 04 02:39:01.070573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.070536 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" event={"ID":"e830410b-0936-4d99-9932-ffa515c1760a","Type":"ContainerStarted","Data":"b6dcccd4fe69ed7ac87f0582f188960592b5508032574a4a9223f0f514f77431"} Nov 04 02:39:01.070772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.070675 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:39:01.226590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.226556 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:01.229971 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.229953 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:01.261992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.261938 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" podStartSLOduration=2.235330892 podStartE2EDuration="6.261923157s" totalImagesPullingTime="4.026592265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:39:01.26108685 +0000 UTC m=+388.227587840" watchObservedRunningTime="2026-11-04 02:39:01.261923157 +0000 UTC m=+388.228424148" Nov 04 02:39:01.283805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.283725 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wpmvk\"" Nov 04 02:39:01.283977 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.283958 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:01.339516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.339482 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbn2\" (UniqueName: \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\") pod \"authorino-57f7f8659f-dcx6s\" (UID: \"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43\") " pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:01.440595 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.440559 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbn2\" (UniqueName: \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\") pod \"authorino-57f7f8659f-dcx6s\" (UID: \"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43\") " pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:01.503028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.502994 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbn2\" (UniqueName: \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\") pod \"authorino-57f7f8659f-dcx6s\" (UID: \"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43\") " pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:01.541646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.541561 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:01.915622 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:01.915598 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:01.916858 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:39:01.916835 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5df09eb_a57b_4a6e_9a88_21a2d95c9d43.slice/crio-b8f4e2fd43be0be3fca34dece5fedf941cd4821234b3049f841d945060a2bb96 WatchSource:0}: Error finding container b8f4e2fd43be0be3fca34dece5fedf941cd4821234b3049f841d945060a2bb96: Status 404 returned error can't find the container with id b8f4e2fd43be0be3fca34dece5fedf941cd4821234b3049f841d945060a2bb96 Nov 04 02:39:02.082075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:02.082027 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" event={"ID":"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43","Type":"ContainerStarted","Data":"b8f4e2fd43be0be3fca34dece5fedf941cd4821234b3049f841d945060a2bb96"} Nov 04 02:39:05.100846 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:05.100811 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" event={"ID":"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43","Type":"ContainerStarted","Data":"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf"} Nov 04 02:39:05.217736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:05.217686 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" podStartSLOduration=1.6174169809999999 podStartE2EDuration="4.217672388s" totalImagesPullingTime="2.600255407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:39:05.216156462 +0000 UTC m=+392.182657457" watchObservedRunningTime="2026-11-04 02:39:05.217672388 +0000 UTC m=+392.184173378" Nov 04 02:39:06.400140 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:39:06.400101 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:39:07.422155 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.422124 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:07.422643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.422317 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" podUID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" containerName="authorino" containerID="cri-o://81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf" gracePeriod=30 Nov 04 02:39:07.672523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.672462 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:07.803649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.803608 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\" (UniqueName: \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\") pod \"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43\" (UID: \"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43\") " Nov 04 02:39:07.805993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.805963 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2" pod "d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" (UID: "d5df09eb-a57b-4a6e-9a88-21a2d95c9d43"). InnerVolumeSpecName "kube-api-access-7mbn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:39:07.904491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:07.904455 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbn2\" (UniqueName: \"kubernetes.io/projected/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43-kube-api-access-7mbn2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:39:08.116960 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.116871 4784 generic.go:356] "Generic (PLEG): container finished" podID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" containerID="81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf" exitCode=0 Nov 04 02:39:08.116960 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.116921 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" Nov 04 02:39:08.117143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.116921 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" event={"ID":"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43","Type":"ContainerDied","Data":"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf"} Nov 04 02:39:08.117143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.117020 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcx6s" event={"ID":"d5df09eb-a57b-4a6e-9a88-21a2d95c9d43","Type":"ContainerDied","Data":"b8f4e2fd43be0be3fca34dece5fedf941cd4821234b3049f841d945060a2bb96"} Nov 04 02:39:08.117143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.117042 4784 scope.go:122] "RemoveContainer" containerID="81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf" Nov 04 02:39:08.125964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.125944 4784 scope.go:122] "RemoveContainer" containerID="81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf" Nov 04 02:39:08.126266 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:39:08.126239 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf\": container with ID starting with 81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf not found: ID does not exist" containerID="81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf" Nov 04 02:39:08.126357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.126273 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf"} err="failed to get container status \"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf\": rpc error: code = NotFound desc = could not find container \"81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf\": container with ID starting with 81fe4cc33c11db8cfd4dc196347afb9961b1dfbb8bc7d9b231cbbab69bfcf7bf not found: ID does not exist" Nov 04 02:39:08.284393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.284356 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:08.289597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:08.289571 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcx6s"] Nov 04 02:39:09.158071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:09.158036 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" path="/var/lib/kubelet/pods/d5df09eb-a57b-4a6e-9a88-21a2d95c9d43/volumes" Nov 04 02:39:14.381727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.381648 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:39:14.382235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.381855 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" podUID="2e223a20-f5c8-430d-9a54-76f78577a543" containerName="manager" containerID="cri-o://e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca" gracePeriod=10 Nov 04 02:39:14.513833 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.513794 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt"] Nov 04 02:39:14.514283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.514265 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" containerName="authorino" Nov 04 02:39:14.514383 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.514284 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" containerName="authorino" Nov 04 02:39:14.514439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.514391 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5df09eb-a57b-4a6e-9a88-21a2d95c9d43" containerName="authorino" Nov 04 02:39:14.517398 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.517376 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.560870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.560835 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnt5n\" (UniqueName: \"kubernetes.io/projected/0f792d27-8080-4b68-b011-04f47f88d35e-kube-api-access-hnt5n\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.561050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.560914 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0f792d27-8080-4b68-b011-04f47f88d35e-manager-config\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.561117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.561057 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-metrics-certs\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.561117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.561096 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-cert\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.573898 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.573870 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 04 02:39:14.573898 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.573881 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 02:39:14.578063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.578011 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4z7fv\"" Nov 04 02:39:14.580597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.580560 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt"] Nov 04 02:39:14.645850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.645790 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:39:14.661979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.661951 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\" (UniqueName: \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\") pod \"2e223a20-f5c8-430d-9a54-76f78577a543\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " Nov 04 02:39:14.662140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662024 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\") pod \"2e223a20-f5c8-430d-9a54-76f78577a543\" (UID: \"2e223a20-f5c8-430d-9a54-76f78577a543\") " Nov 04 02:39:14.662140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662111 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt5n\" (UniqueName: \"kubernetes.io/projected/0f792d27-8080-4b68-b011-04f47f88d35e-kube-api-access-hnt5n\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.662273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662160 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0f792d27-8080-4b68-b011-04f47f88d35e-manager-config\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.662273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662232 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-metrics-certs\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.662273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662256 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-cert\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.662629 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662603 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume" pod "2e223a20-f5c8-430d-9a54-76f78577a543" (UID: "2e223a20-f5c8-430d-9a54-76f78577a543"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:39:14.662933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.662912 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0f792d27-8080-4b68-b011-04f47f88d35e-manager-config\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.664264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.664240 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss" pod "2e223a20-f5c8-430d-9a54-76f78577a543" (UID: "2e223a20-f5c8-430d-9a54-76f78577a543"). InnerVolumeSpecName "kube-api-access-7ghss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:39:14.679254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.679202 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 04 02:39:14.679463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.679322 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:39:14.679463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.679370 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 04 02:39:14.685669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.685644 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-cert\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.685837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.685819 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f792d27-8080-4b68-b011-04f47f88d35e-metrics-certs\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.736342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.736300 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnt5n\" (UniqueName: \"kubernetes.io/projected/0f792d27-8080-4b68-b011-04f47f88d35e-kube-api-access-hnt5n\") pod \"jobset-controller-manager-5d479b4bc7-v87qt\" (UID: \"0f792d27-8080-4b68-b011-04f47f88d35e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:14.763525 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.763494 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghss\" (UniqueName: \"kubernetes.io/projected/2e223a20-f5c8-430d-9a54-76f78577a543-kube-api-access-7ghss\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:39:14.763525 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.763523 4784 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e223a20-f5c8-430d-9a54-76f78577a543-extensions-socket-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:39:14.830936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:14.830906 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:15.152308 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.152272 4784 generic.go:356] "Generic (PLEG): container finished" podID="2e223a20-f5c8-430d-9a54-76f78577a543" containerID="e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca" exitCode=0 Nov 04 02:39:15.152505 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.152325 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" Nov 04 02:39:15.152505 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.152363 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" event={"ID":"2e223a20-f5c8-430d-9a54-76f78577a543","Type":"ContainerDied","Data":"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca"} Nov 04 02:39:15.152505 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.152429 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw" event={"ID":"2e223a20-f5c8-430d-9a54-76f78577a543","Type":"ContainerDied","Data":"e3dbc041f1a9e21692f2b9a1171f8a8b4ca302223e30dfc4565a897f59c6d556"} Nov 04 02:39:15.152505 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.152498 4784 scope.go:122] "RemoveContainer" containerID="e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca" Nov 04 02:39:15.162923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.162905 4784 scope.go:122] "RemoveContainer" containerID="e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca" Nov 04 02:39:15.163201 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:39:15.163170 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca\": container with ID starting with e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca not found: ID does not exist" containerID="e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca" Nov 04 02:39:15.163284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.163211 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca"} err="failed to get container status \"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca\": rpc error: code = NotFound desc = could not find container \"e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca\": container with ID starting with e3f50d24f167e7019837f90afd27429fe9f39f747032ace2d0c735a6671888ca not found: ID does not exist" Nov 04 02:39:15.211836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.211810 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt"] Nov 04 02:39:15.212986 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:39:15.212962 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f792d27_8080_4b68_b011_04f47f88d35e.slice/crio-a7aa379d0fae82ac2f46a30e97ed526162e25df7a16978bdcec7fdf60fe1ed1b WatchSource:0}: Error finding container a7aa379d0fae82ac2f46a30e97ed526162e25df7a16978bdcec7fdf60fe1ed1b: Status 404 returned error can't find the container with id a7aa379d0fae82ac2f46a30e97ed526162e25df7a16978bdcec7fdf60fe1ed1b Nov 04 02:39:15.340431 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.340382 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:39:15.349000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:15.348975 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7qfrw"] Nov 04 02:39:16.159223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:16.159173 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" event={"ID":"0f792d27-8080-4b68-b011-04f47f88d35e","Type":"ContainerStarted","Data":"a7aa379d0fae82ac2f46a30e97ed526162e25df7a16978bdcec7fdf60fe1ed1b"} Nov 04 02:39:17.158837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:17.158801 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e223a20-f5c8-430d-9a54-76f78577a543" path="/var/lib/kubelet/pods/2e223a20-f5c8-430d-9a54-76f78577a543/volumes" Nov 04 02:39:20.183254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:20.183222 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" event={"ID":"0f792d27-8080-4b68-b011-04f47f88d35e","Type":"ContainerStarted","Data":"96875ed6cfe23abf1489cfe2b486061897bd527ccb0177908bfdeacabf2e5a47"} Nov 04 02:39:20.183693 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:20.183366 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:20.304975 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:20.304913 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" podStartSLOduration=1.811832876 podStartE2EDuration="6.304891953s" totalImagesPullingTime="4.493059077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:39:20.302620129 +0000 UTC m=+407.269121132" watchObservedRunningTime="2026-11-04 02:39:20.304891953 +0000 UTC m=+407.271392944" Nov 04 02:39:22.083982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:22.083948 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-k7c2j" Nov 04 02:39:31.193774 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:39:31.193732 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d479b4bc7-v87qt" Nov 04 02:39:36.399337 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:39:36.399297 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:40:01.882037 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.881997 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:01.883647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.883618 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2e223a20-f5c8-430d-9a54-76f78577a543" containerName="manager" Nov 04 02:40:01.885321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.885288 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e223a20-f5c8-430d-9a54-76f78577a543" containerName="manager" Nov 04 02:40:01.885776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.885756 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e223a20-f5c8-430d-9a54-76f78577a543" containerName="manager" Nov 04 02:40:01.895389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.895355 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:01.945457 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.945420 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:01.993356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.993320 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:01.993356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.993360 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:01.993575 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.993468 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkwz\" (UniqueName: \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:01.994182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.994150 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 02:40:01.994307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.994156 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 02:40:01.994307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.994157 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7mq2\"" Nov 04 02:40:01.994307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:01.994263 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 02:40:02.094718 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.094682 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.094897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.094784 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkwz\" (UniqueName: \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.094897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.094839 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.097539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.097508 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.097539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.097532 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.158652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.158570 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkwz\" (UniqueName: \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\") pod \"maas-controller-8b95c9c66-nprd9\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.208531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.208497 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:02.314494 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.314455 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 02:40:02.319203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.319170 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.379349 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.379301 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 02:40:02.398230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.398176 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.398560 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.398538 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bpl\" (UniqueName: \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.398845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.398794 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.500075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.500030 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.500272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.500113 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bpl\" (UniqueName: \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.500272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.500149 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.502675 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.502651 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.502774 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.502680 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.562633 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.562595 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bpl\" (UniqueName: \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\") pod \"maas-controller-78787b6955-88mxk\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:02.587933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.587902 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:02.589337 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:02.589310 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b59121_2fd9_48b0_a094_fbab0e334e85.slice/crio-d6451b6c9cdef86d17061a8bacc0bae360a0a14a8a1b448d0002d4f5a5defd52 WatchSource:0}: Error finding container d6451b6c9cdef86d17061a8bacc0bae360a0a14a8a1b448d0002d4f5a5defd52: Status 404 returned error can't find the container with id d6451b6c9cdef86d17061a8bacc0bae360a0a14a8a1b448d0002d4f5a5defd52 Nov 04 02:40:02.632822 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:02.632792 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:03.012624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:03.012588 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 02:40:03.013680 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:03.013647 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode814c148_d53f_4f7c_9728_81d1f504652b.slice/crio-ea5a45fd9057b6515e9b5094362f400634b3367cb29229a7480b12f32dbcf514 WatchSource:0}: Error finding container ea5a45fd9057b6515e9b5094362f400634b3367cb29229a7480b12f32dbcf514: Status 404 returned error can't find the container with id ea5a45fd9057b6515e9b5094362f400634b3367cb29229a7480b12f32dbcf514 Nov 04 02:40:03.412053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:03.411960 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-88mxk" event={"ID":"e814c148-d53f-4f7c-9728-81d1f504652b","Type":"ContainerStarted","Data":"ea5a45fd9057b6515e9b5094362f400634b3367cb29229a7480b12f32dbcf514"} Nov 04 02:40:03.413145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:03.413120 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-nprd9" event={"ID":"54b59121-2fd9-48b0-a094-fbab0e334e85","Type":"ContainerStarted","Data":"d6451b6c9cdef86d17061a8bacc0bae360a0a14a8a1b448d0002d4f5a5defd52"} Nov 04 02:40:04.948467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:04.948428 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dbq49"] Nov 04 02:40:04.959708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:04.959674 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.008953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.008883 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dbq49"] Nov 04 02:40:05.014866 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.014175 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s7528\"" Nov 04 02:40:05.014866 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.014286 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 02:40:05.023899 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.023860 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvcv\" (UniqueName: \"kubernetes.io/projected/c4d067ed-305b-40ee-bb99-3c3cab436dab-kube-api-access-4bvcv\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.024092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.024060 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-cert\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.024163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.024116 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-metrics-certs\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.057628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.057573 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 02:40:05.124925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.124887 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-metrics-certs\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.125112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.124964 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvcv\" (UniqueName: \"kubernetes.io/projected/c4d067ed-305b-40ee-bb99-3c3cab436dab-kube-api-access-4bvcv\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.125112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.125037 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-cert\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.131609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.131577 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-cert\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.145079 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.145052 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d067ed-305b-40ee-bb99-3c3cab436dab-metrics-certs\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.190135 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.190100 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvcv\" (UniqueName: \"kubernetes.io/projected/c4d067ed-305b-40ee-bb99-3c3cab436dab-kube-api-access-4bvcv\") pod \"odh-model-controller-769875db56-dbq49\" (UID: \"c4d067ed-305b-40ee-bb99-3c3cab436dab\") " pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.277955 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.277875 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:05.673367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:05.673335 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dbq49"] Nov 04 02:40:05.674567 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:05.674535 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d067ed_305b_40ee_bb99_3c3cab436dab.slice/crio-4f67017f41a957500ced778b6295ca0d0499f08ecd32d04f73799606e50204f6 WatchSource:0}: Error finding container 4f67017f41a957500ced778b6295ca0d0499f08ecd32d04f73799606e50204f6: Status 404 returned error can't find the container with id 4f67017f41a957500ced778b6295ca0d0499f08ecd32d04f73799606e50204f6 Nov 04 02:40:06.399308 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:40:06.399278 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:40:06.439122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:06.439089 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dbq49" event={"ID":"c4d067ed-305b-40ee-bb99-3c3cab436dab","Type":"ContainerStarted","Data":"4f67017f41a957500ced778b6295ca0d0499f08ecd32d04f73799606e50204f6"} Nov 04 02:40:06.440727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:06.440698 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-nprd9" event={"ID":"54b59121-2fd9-48b0-a094-fbab0e334e85","Type":"ContainerStarted","Data":"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28"} Nov 04 02:40:06.440855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:06.440840 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:06.575510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:06.575453 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8b95c9c66-nprd9" podStartSLOduration=1.962030726 podStartE2EDuration="5.575439064s" totalImagesPullingTime="3.613408338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:06.573727105 +0000 UTC m=+453.540228098" watchObservedRunningTime="2026-11-04 02:40:06.575439064 +0000 UTC m=+453.541940055" Nov 04 02:40:07.448646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:07.448601 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-88mxk" event={"ID":"e814c148-d53f-4f7c-9728-81d1f504652b","Type":"ContainerStarted","Data":"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92"} Nov 04 02:40:07.570399 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:07.570034 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78787b6955-88mxk" podStartSLOduration=2.186654601 podStartE2EDuration="5.570013111s" totalImagesPullingTime="3.38335851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:07.569366006 +0000 UTC m=+454.535867038" watchObservedRunningTime="2026-11-04 02:40:07.570013111 +0000 UTC m=+454.536514103" Nov 04 02:40:08.454694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:08.454657 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:09.468562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:09.468520 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dbq49" event={"ID":"c4d067ed-305b-40ee-bb99-3c3cab436dab","Type":"ContainerStarted","Data":"9a2c2ca542905379bec3e7108f4c141a9a9395423995d6d803672bd8e18b2928"} Nov 04 02:40:09.469032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:09.468766 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:09.587232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:09.587152 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dbq49" podStartSLOduration=2.360710138 podStartE2EDuration="5.587140111s" totalImagesPullingTime="3.226429973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:09.586857673 +0000 UTC m=+456.553358666" watchObservedRunningTime="2026-11-04 02:40:09.587140111 +0000 UTC m=+456.553641100" Nov 04 02:40:17.451567 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:17.451537 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:19.471027 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.470992 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 02:40:19.644553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.644515 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:19.644797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.644770 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8b95c9c66-nprd9" podUID="54b59121-2fd9-48b0-a094-fbab0e334e85" containerName="manager" containerID="cri-o://cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28" gracePeriod=30 Nov 04 02:40:19.911295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.911270 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:19.975159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.975124 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\") pod \"54b59121-2fd9-48b0-a094-fbab0e334e85\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " Nov 04 02:40:19.975159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.975171 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\") pod \"54b59121-2fd9-48b0-a094-fbab0e334e85\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " Nov 04 02:40:19.975469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.975267 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\" (UniqueName: \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\") pod \"54b59121-2fd9-48b0-a094-fbab0e334e85\" (UID: \"54b59121-2fd9-48b0-a094-fbab0e334e85\") " Nov 04 02:40:19.977678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.977639 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs" pod "54b59121-2fd9-48b0-a094-fbab0e334e85" (UID: "54b59121-2fd9-48b0-a094-fbab0e334e85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:40:19.977678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.977664 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert" pod "54b59121-2fd9-48b0-a094-fbab0e334e85" (UID: "54b59121-2fd9-48b0-a094-fbab0e334e85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:40:19.977835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:19.977700 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz" pod "54b59121-2fd9-48b0-a094-fbab0e334e85" (UID: "54b59121-2fd9-48b0-a094-fbab0e334e85"). InnerVolumeSpecName "kube-api-access-2qkwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:40:20.076021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.075926 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkwz\" (UniqueName: \"kubernetes.io/projected/54b59121-2fd9-48b0-a094-fbab0e334e85-kube-api-access-2qkwz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:40:20.076021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.075963 4784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-metrics-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:40:20.076021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.075977 4784 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54b59121-2fd9-48b0-a094-fbab0e334e85-webhook-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:40:20.477065 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.477031 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dbq49" Nov 04 02:40:20.525222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.525169 4784 generic.go:356] "Generic (PLEG): container finished" podID="54b59121-2fd9-48b0-a094-fbab0e334e85" containerID="cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28" exitCode=0 Nov 04 02:40:20.525393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.525258 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-nprd9" Nov 04 02:40:20.525393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.525255 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-nprd9" event={"ID":"54b59121-2fd9-48b0-a094-fbab0e334e85","Type":"ContainerDied","Data":"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28"} Nov 04 02:40:20.525393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.525382 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-nprd9" event={"ID":"54b59121-2fd9-48b0-a094-fbab0e334e85","Type":"ContainerDied","Data":"d6451b6c9cdef86d17061a8bacc0bae360a0a14a8a1b448d0002d4f5a5defd52"} Nov 04 02:40:20.525498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.525405 4784 scope.go:122] "RemoveContainer" containerID="cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28" Nov 04 02:40:20.535008 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.534986 4784 scope.go:122] "RemoveContainer" containerID="cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28" Nov 04 02:40:20.535335 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:40:20.535317 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28\": container with ID starting with cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28 not found: ID does not exist" containerID="cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28" Nov 04 02:40:20.535392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.535345 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28"} err="failed to get container status \"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28\": rpc error: code = NotFound desc = could not find container \"cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28\": container with ID starting with cce4131c7e5188af5056b6f7c7e1359d494ee2e2601f08d6cbda5e69cb3a4e28 not found: ID does not exist" Nov 04 02:40:20.710602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.710560 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:20.716433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:20.716405 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-nprd9"] Nov 04 02:40:21.159032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:21.158994 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b59121-2fd9-48b0-a094-fbab0e334e85" path="/var/lib/kubelet/pods/54b59121-2fd9-48b0-a094-fbab0e334e85/volumes" Nov 04 02:40:36.400866 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:40:36.400831 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:40:55.470358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.470314 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:40:55.470887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.470869 4784 cpu_manager.go:414] "containerMap: removing container" podUID="54b59121-2fd9-48b0-a094-fbab0e334e85" containerName="manager" Nov 04 02:40:55.470929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.470890 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b59121-2fd9-48b0-a094-fbab0e334e85" containerName="manager" Nov 04 02:40:55.471044 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.471033 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b59121-2fd9-48b0-a094-fbab0e334e85" containerName="manager" Nov 04 02:40:55.473766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.473748 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:55.512783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.512748 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkk6\" (UniqueName: \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\") pod \"authorino-6f6944bcdd-6p8lm\" (UID: \"b687a116-a73e-4fa4-ac77-9bc9a703e4f3\") " pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:55.529331 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.529305 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:40:55.577606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.577582 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wpmvk\"" Nov 04 02:40:55.613688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.613651 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkk6\" (UniqueName: \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\") pod \"authorino-6f6944bcdd-6p8lm\" (UID: \"b687a116-a73e-4fa4-ac77-9bc9a703e4f3\") " pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:55.676075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.676045 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkk6\" (UniqueName: \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\") pod \"authorino-6f6944bcdd-6p8lm\" (UID: \"b687a116-a73e-4fa4-ac77-9bc9a703e4f3\") " pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:55.784683 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.784648 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:55.829043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.829008 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:40:55.853995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.853903 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:40:55.856945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.856600 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:55.913050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.913017 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:40:55.916427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:55.916396 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpvcz\" (UniqueName: \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\") pod \"authorino-7b8f8b78b9-9f8hb\" (UID: \"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93\") " pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:56.017594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.017502 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpvcz\" (UniqueName: \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\") pod \"authorino-7b8f8b78b9-9f8hb\" (UID: \"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93\") " pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:56.080317 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.080277 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpvcz\" (UniqueName: \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\") pod \"authorino-7b8f8b78b9-9f8hb\" (UID: \"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93\") " pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:56.157235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.157207 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:40:56.159388 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:56.159360 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb687a116_a73e_4fa4_ac77_9bc9a703e4f3.slice/crio-e51109c3c3b059f05aa0d0bad5964880ab656165c0620a34ab06d2c8dd33eb1d WatchSource:0}: Error finding container e51109c3c3b059f05aa0d0bad5964880ab656165c0620a34ab06d2c8dd33eb1d: Status 404 returned error can't find the container with id e51109c3c3b059f05aa0d0bad5964880ab656165c0620a34ab06d2c8dd33eb1d Nov 04 02:40:56.168932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.168908 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:56.293396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.293308 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:40:56.323832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.323794 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:40:56.326144 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.326121 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.381721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.381690 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 02:40:56.384106 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.384080 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:40:56.420316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.420279 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.420316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.420323 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dz7\" (UniqueName: \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.521370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.521321 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dz7\" (UniqueName: \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.521886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.521427 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.524018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.523994 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.548418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.548342 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:40:56.548418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.548392 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:40:56.549156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.549132 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:40:56.551220 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:56.551179 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14fb5d1e_3a15_4c1b_a26d_3c7b6b1ebc93.slice/crio-a18166d1ee02144f6496994f9c0604544c2a124a39ebfd6f7d429a1a470a1b5c WatchSource:0}: Error finding container a18166d1ee02144f6496994f9c0604544c2a124a39ebfd6f7d429a1a470a1b5c: Status 404 returned error can't find the container with id a18166d1ee02144f6496994f9c0604544c2a124a39ebfd6f7d429a1a470a1b5c Nov 04 02:40:56.584148 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.584107 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dz7\" (UniqueName: \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\") pod \"authorino-c64447b99-b8bpz\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.639222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.639164 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:40:56.716930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.716892 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" event={"ID":"b687a116-a73e-4fa4-ac77-9bc9a703e4f3","Type":"ContainerStarted","Data":"e51109c3c3b059f05aa0d0bad5964880ab656165c0620a34ab06d2c8dd33eb1d"} Nov 04 02:40:56.718534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:56.718499 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" event={"ID":"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93","Type":"ContainerStarted","Data":"a18166d1ee02144f6496994f9c0604544c2a124a39ebfd6f7d429a1a470a1b5c"} Nov 04 02:40:57.016719 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:57.016691 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:40:57.019165 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:40:57.019131 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4d36e3_19f3_4117_bf5b_69e99e69398a.slice/crio-04e974261668a1d3824bf217743fe32750739cf2d45501ebe71c1d4804843664 WatchSource:0}: Error finding container 04e974261668a1d3824bf217743fe32750739cf2d45501ebe71c1d4804843664: Status 404 returned error can't find the container with id 04e974261668a1d3824bf217743fe32750739cf2d45501ebe71c1d4804843664 Nov 04 02:40:57.726691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:57.726655 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c64447b99-b8bpz" event={"ID":"4c4d36e3-19f3-4117-bf5b-69e99e69398a","Type":"ContainerStarted","Data":"04e974261668a1d3824bf217743fe32750739cf2d45501ebe71c1d4804843664"} Nov 04 02:40:58.734126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.734025 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" event={"ID":"b687a116-a73e-4fa4-ac77-9bc9a703e4f3","Type":"ContainerStarted","Data":"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2"} Nov 04 02:40:58.734126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.734065 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" podUID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" containerName="authorino" containerID="cri-o://21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2" gracePeriod=30 Nov 04 02:40:58.736146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.736116 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" event={"ID":"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93","Type":"ContainerStarted","Data":"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4"} Nov 04 02:40:58.736312 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.736174 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" podUID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" containerName="authorino" containerID="cri-o://44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4" gracePeriod=30 Nov 04 02:40:58.738173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.738148 4784 generic.go:356] "Generic (PLEG): container finished" podID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerID="569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c" exitCode=0 Nov 04 02:40:58.738316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.738253 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerDied","Data":"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c"} Nov 04 02:40:58.738703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.738682 4784 scope.go:122] "RemoveContainer" containerID="569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c" Nov 04 02:40:58.741043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.739836 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c64447b99-b8bpz" event={"ID":"4c4d36e3-19f3-4117-bf5b-69e99e69398a","Type":"ContainerStarted","Data":"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e"} Nov 04 02:40:58.741043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.740658 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:40:58.741043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.740728 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:40:58.741043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.740759 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:40:58.853545 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:58.852648 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" podStartSLOduration=2.166614993 podStartE2EDuration="3.852631196s" totalImagesPullingTime="1.686016203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:58.851310139 +0000 UTC m=+505.817811131" watchObservedRunningTime="2026-11-04 02:40:58.852631196 +0000 UTC m=+505.819132189" Nov 04 02:40:59.078358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.078324 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:59.083044 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.082966 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:59.102869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.102800 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" podStartSLOduration=2.523693109 podStartE2EDuration="4.10278028s" totalImagesPullingTime="1.579087171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:59.102151891 +0000 UTC m=+506.068652894" watchObservedRunningTime="2026-11-04 02:40:59.10278028 +0000 UTC m=+506.069281274" Nov 04 02:40:59.152734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.152619 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\" (UniqueName: \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\") pod \"b687a116-a73e-4fa4-ac77-9bc9a703e4f3\" (UID: \"b687a116-a73e-4fa4-ac77-9bc9a703e4f3\") " Nov 04 02:40:59.152734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.152721 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\" (UniqueName: \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\") pod \"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93\" (UID: \"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93\") " Nov 04 02:40:59.155929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.155896 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6" pod "b687a116-a73e-4fa4-ac77-9bc9a703e4f3" (UID: "b687a116-a73e-4fa4-ac77-9bc9a703e4f3"). InnerVolumeSpecName "kube-api-access-6bkk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:40:59.156091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.156050 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz" pod "14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" (UID: "14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93"). InnerVolumeSpecName "kube-api-access-wpvcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:40:59.220859 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.220806 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c64447b99-b8bpz" podStartSLOduration=1.8857170829999998 podStartE2EDuration="3.220791387s" totalImagesPullingTime="1.335074304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:59.218991811 +0000 UTC m=+506.185492803" watchObservedRunningTime="2026-11-04 02:40:59.220791387 +0000 UTC m=+506.187292453" Nov 04 02:40:59.256147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.253884 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkk6\" (UniqueName: \"kubernetes.io/projected/b687a116-a73e-4fa4-ac77-9bc9a703e4f3-kube-api-access-6bkk6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:40:59.256147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.253930 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpvcz\" (UniqueName: \"kubernetes.io/projected/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93-kube-api-access-wpvcz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:40:59.746001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.745964 4784 generic.go:356] "Generic (PLEG): container finished" podID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" containerID="21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2" exitCode=0 Nov 04 02:40:59.746482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.746008 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" Nov 04 02:40:59.746482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.746041 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" event={"ID":"b687a116-a73e-4fa4-ac77-9bc9a703e4f3","Type":"ContainerDied","Data":"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2"} Nov 04 02:40:59.746482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.746084 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-6p8lm" event={"ID":"b687a116-a73e-4fa4-ac77-9bc9a703e4f3","Type":"ContainerDied","Data":"e51109c3c3b059f05aa0d0bad5964880ab656165c0620a34ab06d2c8dd33eb1d"} Nov 04 02:40:59.746482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.746104 4784 scope.go:122] "RemoveContainer" containerID="21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2" Nov 04 02:40:59.747437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.747403 4784 generic.go:356] "Generic (PLEG): container finished" podID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" containerID="44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4" exitCode=0 Nov 04 02:40:59.747528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.747504 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" event={"ID":"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93","Type":"ContainerDied","Data":"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4"} Nov 04 02:40:59.747647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.747544 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" event={"ID":"14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93","Type":"ContainerDied","Data":"a18166d1ee02144f6496994f9c0604544c2a124a39ebfd6f7d429a1a470a1b5c"} Nov 04 02:40:59.747647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.747503 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8f8b78b9-9f8hb" Nov 04 02:40:59.749875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.749802 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerStarted","Data":"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c"} Nov 04 02:40:59.786885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.781595 4784 scope.go:122] "RemoveContainer" containerID="21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2" Nov 04 02:40:59.786885 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:40:59.782082 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2\": container with ID starting with 21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2 not found: ID does not exist" containerID="21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2" Nov 04 02:40:59.786885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.782124 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2"} err="failed to get container status \"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2\": rpc error: code = NotFound desc = could not find container \"21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2\": container with ID starting with 21dde2f7e78217f9f97d200943d442ef6720874f713953613f59ad0961337ff2 not found: ID does not exist" Nov 04 02:40:59.786885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.782150 4784 scope.go:122] "RemoveContainer" containerID="44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4" Nov 04 02:40:59.795130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.795107 4784 scope.go:122] "RemoveContainer" containerID="44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4" Nov 04 02:40:59.795548 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:40:59.795528 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4\": container with ID starting with 44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4 not found: ID does not exist" containerID="44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4" Nov 04 02:40:59.795641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:40:59.795559 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4"} err="failed to get container status \"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4\": rpc error: code = NotFound desc = could not find container \"44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4\": container with ID starting with 44a364982c065a33d65187ee9fddff3af44d55e1d44f99d4e6a67d5e294641e4 not found: ID does not exist" Nov 04 02:41:00.041108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.041010 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:41:00.050714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.050677 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b8f8b78b9-9f8hb"] Nov 04 02:41:00.173338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.173262 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:41:00.176738 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.176675 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6p8lm"] Nov 04 02:41:00.715100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.715064 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:41:00.715364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.715338 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 04 02:41:00.715445 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:00.715395 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 04 02:41:01.182725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:01.182687 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" path="/var/lib/kubelet/pods/14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93/volumes" Nov 04 02:41:01.183275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:01.183060 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" path="/var/lib/kubelet/pods/b687a116-a73e-4fa4-ac77-9bc9a703e4f3/volumes" Nov 04 02:41:01.715310 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:01.715272 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 04 02:41:01.715491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:01.715331 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 04 02:41:02.715484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:02.715447 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 04 02:41:02.715890 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:02.715513 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 04 02:41:03.715406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:03.715370 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 04 02:41:03.715664 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:03.715431 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 04 02:41:04.715811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:04.715768 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 04 02:41:04.716348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:04.715868 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 04 02:41:05.718746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:05.718713 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:41:05.719291 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:05.719087 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:41:05.719695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:41:05.719676 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 02:41:06.399166 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:41:06.399130 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:41:36.398829 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:41:36.398784 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:42:06.399328 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:42:06.399273 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:42:29.664246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664208 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw"] Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664574 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" containerName="authorino" Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664584 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" containerName="authorino" Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664601 4784 cpu_manager.go:414] "containerMap: removing container" podUID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" containerName="authorino" Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664607 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" containerName="authorino" Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664674 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b687a116-a73e-4fa4-ac77-9bc9a703e4f3" containerName="authorino" Nov 04 02:42:29.664699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.664683 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fb5d1e-3a15-4c1b-a26d-3c7b6b1ebc93" containerName="authorino" Nov 04 02:42:29.667609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.667592 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678367 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678497 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678411 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzgxb\" (UniqueName: \"kubernetes.io/projected/27aed879-c829-4755-98ea-9d12676f9bf5-kube-api-access-nzgxb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678497 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678491 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27aed879-c829-4755-98ea-9d12676f9bf5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678568 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678553 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678596 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.678645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.678619 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.715795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.715759 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw"] Nov 04 02:42:29.779514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779481 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779528 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779551 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779599 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779632 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzgxb\" (UniqueName: \"kubernetes.io/projected/27aed879-c829-4755-98ea-9d12676f9bf5-kube-api-access-nzgxb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779683 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27aed879-c829-4755-98ea-9d12676f9bf5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.779995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.779968 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.780062 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.780021 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.780611 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.780086 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.788572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.788545 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27aed879-c829-4755-98ea-9d12676f9bf5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.816048 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.816018 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w44xl\"" Nov 04 02:42:29.823540 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.823518 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 02:42:29.833161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.833125 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27aed879-c829-4755-98ea-9d12676f9bf5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.867173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.867131 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 02:42:29.867173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.867182 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 02:42:29.928833 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.928760 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzgxb\" (UniqueName: \"kubernetes.io/projected/27aed879-c829-4755-98ea-9d12676f9bf5-kube-api-access-nzgxb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw\" (UID: \"27aed879-c829-4755-98ea-9d12676f9bf5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:29.980462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:29.980412 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:30.364108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:30.364062 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw"] Nov 04 02:42:30.367161 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:42:30.367132 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27aed879_c829_4755_98ea_9d12676f9bf5.slice/crio-92336c0a1b2395d48e5098bacf61d32eb14739cf18e14a3988009a7bbe066e1a WatchSource:0}: Error finding container 92336c0a1b2395d48e5098bacf61d32eb14739cf18e14a3988009a7bbe066e1a: Status 404 returned error can't find the container with id 92336c0a1b2395d48e5098bacf61d32eb14739cf18e14a3988009a7bbe066e1a Nov 04 02:42:31.240146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:31.240099 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" event={"ID":"27aed879-c829-4755-98ea-9d12676f9bf5","Type":"ContainerStarted","Data":"92336c0a1b2395d48e5098bacf61d32eb14739cf18e14a3988009a7bbe066e1a"} Nov 04 02:42:32.247076 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:32.247042 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" event={"ID":"27aed879-c829-4755-98ea-9d12676f9bf5","Type":"ContainerStarted","Data":"cc95b0411aec61686e3decb265c08662ddef7e0358a8813d3e1ef4485f13fceb"} Nov 04 02:42:32.420044 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:32.419982 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" podStartSLOduration=5.908952997 podStartE2EDuration="7.419968687s" totalImagesPullingTime="1.51101569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:32.367432608 +0000 UTC m=+599.333933599" watchObservedRunningTime="2026-11-04 02:42:32.419968687 +0000 UTC m=+599.386469679" Nov 04 02:42:32.981422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:32.981391 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:33.334027 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:33.333942 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:42:33.337586 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:33.337561 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:42:33.339565 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:33.339545 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:42:33.342703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:33.342684 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:42:35.331733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.331693 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc"] Nov 04 02:42:35.335759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.335738 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.426985 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnhh\" (UniqueName: \"kubernetes.io/projected/9322be64-e971-47c8-aa8c-7b513cd2d4d9-kube-api-access-mtnhh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.427048 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.427180 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.427269 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.427383 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.427506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.427430 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.442936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.442906 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 02:42:35.445088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.445068 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc"] Nov 04 02:42:35.528114 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528079 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528114 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528117 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528141 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528218 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnhh\" (UniqueName: \"kubernetes.io/projected/9322be64-e971-47c8-aa8c-7b513cd2d4d9-kube-api-access-mtnhh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528247 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528274 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528537 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528606 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.528727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.528648 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.530694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.530671 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322be64-e971-47c8-aa8c-7b513cd2d4d9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.530833 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.530819 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322be64-e971-47c8-aa8c-7b513cd2d4d9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.612531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.612455 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnhh\" (UniqueName: \"kubernetes.io/projected/9322be64-e971-47c8-aa8c-7b513cd2d4d9-kube-api-access-mtnhh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc\" (UID: \"9322be64-e971-47c8-aa8c-7b513cd2d4d9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:35.649593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:35.649540 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:36.029490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:36.029461 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc"] Nov 04 02:42:36.031144 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:42:36.031114 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9322be64_e971_47c8_aa8c_7b513cd2d4d9.slice/crio-0769588a9d300aea8c89315825f36243c73ec2b8836544072402308d765347b1 WatchSource:0}: Error finding container 0769588a9d300aea8c89315825f36243c73ec2b8836544072402308d765347b1: Status 404 returned error can't find the container with id 0769588a9d300aea8c89315825f36243c73ec2b8836544072402308d765347b1 Nov 04 02:42:36.269520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:36.269474 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" event={"ID":"9322be64-e971-47c8-aa8c-7b513cd2d4d9","Type":"ContainerStarted","Data":"0769588a9d300aea8c89315825f36243c73ec2b8836544072402308d765347b1"} Nov 04 02:42:36.398924 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:42:36.398844 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:42:37.276089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:37.276051 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" event={"ID":"9322be64-e971-47c8-aa8c-7b513cd2d4d9","Type":"ContainerStarted","Data":"29b7d3501e71d43c17448421c11bff61bbdbb2bbb7ba0e1a7c70127f6bb86806"} Nov 04 02:42:37.570026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:37.569924 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" podStartSLOduration=1.998167181 podStartE2EDuration="2.56991085s" totalImagesPullingTime="571.743669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:37.569565575 +0000 UTC m=+604.536066569" watchObservedRunningTime="2026-11-04 02:42:37.56991085 +0000 UTC m=+604.536411841" Nov 04 02:42:37.650316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:37.650276 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:39.722180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.722139 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd"] Nov 04 02:42:39.725806 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.725788 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772057 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772286 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772138 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772294 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772329 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772367 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.772521 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.772498 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5zf\" (UniqueName: \"kubernetes.io/projected/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-kube-api-access-sb5zf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.833065 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.833033 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 02:42:39.836603 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.836569 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd"] Nov 04 02:42:39.873915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.873870 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.873929 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.873973 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874039 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5zf\" (UniqueName: \"kubernetes.io/projected/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-kube-api-access-sb5zf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874077 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874130 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874466 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874541 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.874685 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.874651 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.876607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.876584 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.876783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.876762 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.953747 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.953712 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5zf\" (UniqueName: \"kubernetes.io/projected/8e451643-d5cb-4de9-9b6e-f4d7ce68e625-kube-api-access-sb5zf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rfdbd\" (UID: \"8e451643-d5cb-4de9-9b6e-f4d7ce68e625\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:39.980757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.980652 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:39.994240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:39.994207 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:40.041602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:40.041557 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:40.308949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:40.308871 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw" Nov 04 02:42:40.430295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:40.430261 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd"] Nov 04 02:42:40.430956 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:42:40.430931 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e451643_d5cb_4de9_9b6e_f4d7ce68e625.slice/crio-3a2f5874e5ea07c5eb17f2e1d639f29b7746519f8d45ba28b0173c3bc74d6f5d WatchSource:0}: Error finding container 3a2f5874e5ea07c5eb17f2e1d639f29b7746519f8d45ba28b0173c3bc74d6f5d: Status 404 returned error can't find the container with id 3a2f5874e5ea07c5eb17f2e1d639f29b7746519f8d45ba28b0173c3bc74d6f5d Nov 04 02:42:40.432413 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:40.432398 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:42:41.303030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:41.302940 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" event={"ID":"8e451643-d5cb-4de9-9b6e-f4d7ce68e625","Type":"ContainerStarted","Data":"2445f3dfb6c5f36ff5ba84e66f8e912b76f81955aa0891984e7bf2753698b75f"} Nov 04 02:42:41.303030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:41.302999 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" event={"ID":"8e451643-d5cb-4de9-9b6e-f4d7ce68e625","Type":"ContainerStarted","Data":"3a2f5874e5ea07c5eb17f2e1d639f29b7746519f8d45ba28b0173c3bc74d6f5d"} Nov 04 02:42:41.473704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:41.473656 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" podStartSLOduration=1.894522778 podStartE2EDuration="2.473641039s" totalImagesPullingTime="579.118261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:41.473347606 +0000 UTC m=+608.439848598" watchObservedRunningTime="2026-11-04 02:42:41.473641039 +0000 UTC m=+608.440142030" Nov 04 02:42:42.041949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:42.041910 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:45.650359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:45.650320 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:45.664607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:45.664575 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:46.343130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:46.343097 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc" Nov 04 02:42:50.042598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.042559 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:50.055110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.055083 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:50.363779 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.363696 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rfdbd" Nov 04 02:42:50.423481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.423447 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f"] Nov 04 02:42:50.429320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.429299 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.472680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472642 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.472865 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472690 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.472865 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472812 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195055f3-cf50-4480-b49b-e185904110d8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.472979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472865 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.472979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472936 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.473089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.472995 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd7s\" (UniqueName: \"kubernetes.io/projected/195055f3-cf50-4480-b49b-e185904110d8-kube-api-access-nzd7s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.483713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.483686 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 02:42:50.539504 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.539475 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f"] Nov 04 02:42:50.574071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574039 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574080 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574110 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195055f3-cf50-4480-b49b-e185904110d8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574306 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574349 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574400 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd7s\" (UniqueName: \"kubernetes.io/projected/195055f3-cf50-4480-b49b-e185904110d8-kube-api-access-nzd7s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574548 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574590 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.574683 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.574663 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.576676 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.576651 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195055f3-cf50-4480-b49b-e185904110d8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.577000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.576979 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195055f3-cf50-4480-b49b-e185904110d8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.637384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.637305 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd7s\" (UniqueName: \"kubernetes.io/projected/195055f3-cf50-4480-b49b-e185904110d8-kube-api-access-nzd7s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f\" (UID: \"195055f3-cf50-4480-b49b-e185904110d8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:50.742731 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:50.742693 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:42:51.117590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:51.117561 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f"] Nov 04 02:42:51.118561 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:42:51.118535 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195055f3_cf50_4480_b49b_e185904110d8.slice/crio-e886c900e54dafa57da8688c346f2eb748aaa962aee40b2db9e79c1036b72fbe WatchSource:0}: Error finding container e886c900e54dafa57da8688c346f2eb748aaa962aee40b2db9e79c1036b72fbe: Status 404 returned error can't find the container with id e886c900e54dafa57da8688c346f2eb748aaa962aee40b2db9e79c1036b72fbe Nov 04 02:42:51.359078 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:51.359044 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" event={"ID":"195055f3-cf50-4480-b49b-e185904110d8","Type":"ContainerStarted","Data":"e886c900e54dafa57da8688c346f2eb748aaa962aee40b2db9e79c1036b72fbe"} Nov 04 02:42:52.365788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:52.365755 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" event={"ID":"195055f3-cf50-4480-b49b-e185904110d8","Type":"ContainerStarted","Data":"5b6edaf3fdf22816c9e4396ab0c105e77c01a8b7f02c2e2044cc1ba7b5618b65"} Nov 04 02:42:52.537532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:52.537477 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" podStartSLOduration=1.9548197950000001 podStartE2EDuration="2.537462152s" totalImagesPullingTime="582.642357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:52.485576135 +0000 UTC m=+619.452077128" watchObservedRunningTime="2026-11-04 02:42:52.537462152 +0000 UTC m=+619.503963185" Nov 04 02:42:52.742990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:42:52.742947 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:43:00.136313 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.136280 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:43:00.140050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.140032 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.247949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.247913 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:43:00.263644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.263607 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.263813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.263727 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq4b\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.263813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.263767 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.263813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.263795 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.264001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.263970 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.264042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.264029 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.264177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.264161 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.264261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.264246 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.264326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.264308 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.365867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365826 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.365867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365861 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365885 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365921 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365951 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.365979 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366071 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366176 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366258 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq4b\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366302 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366387 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366613 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366581 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366638 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.366839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.366829 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.368465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.368445 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.368814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.368794 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.428870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.428835 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.429034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.428943 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq4b\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\") pod \"maas-default-gateway-openshift-default-76597d6855-qx6f2\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.452715 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.452681 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:00.743534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.743441 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:43:00.756627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.756598 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:43:00.885900 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:43:00.885868 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9d711a_e05a_4165_bf62_ff1ea5deaa34.slice/crio-5cf0432ad72dfa1199ae122de3bc10296f104b3ca2fb59fd41e1a1009fa870d1 WatchSource:0}: Error finding container 5cf0432ad72dfa1199ae122de3bc10296f104b3ca2fb59fd41e1a1009fa870d1: Status 404 returned error can't find the container with id 5cf0432ad72dfa1199ae122de3bc10296f104b3ca2fb59fd41e1a1009fa870d1 Nov 04 02:43:00.888052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.888016 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:43:00.888115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.888083 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:43:00.888156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.888144 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:43:00.931002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:00.930970 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:43:01.417928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.417892 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" event={"ID":"fd9d711a-e05a-4165-bf62-ff1ea5deaa34","Type":"ContainerStarted","Data":"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93"} Nov 04 02:43:01.418464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.417948 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" event={"ID":"fd9d711a-e05a-4165-bf62-ff1ea5deaa34","Type":"ContainerStarted","Data":"5cf0432ad72dfa1199ae122de3bc10296f104b3ca2fb59fd41e1a1009fa870d1"} Nov 04 02:43:01.429316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.429292 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f" Nov 04 02:43:01.453578 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.453546 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:01.455030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.455002 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:01.455132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.455053 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:01.650168 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:01.650107 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podStartSLOduration=1.650092353 podStartE2EDuration="1.650092353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:01.598118981 +0000 UTC m=+628.564620002" watchObservedRunningTime="2026-11-04 02:43:01.650092353 +0000 UTC m=+628.616593344" Nov 04 02:43:02.453200 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:02.453162 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:02.453615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:02.453239 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:03.454351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:03.454154 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:03.454351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:03.454242 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:04.454112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:04.454072 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:04.454310 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:04.454131 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:05.453944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:05.453909 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:05.454543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:05.453963 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:06.406164 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:43:06.406127 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:43:06.453305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:06.453276 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:06.453484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:06.453327 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:07.453775 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:07.453742 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:07.454285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:07.453798 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:08.453705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:08.453669 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qx6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:43:08.453863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:08.453760 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 04 02:43:09.457899 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:09.457866 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:09.458436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:09.458292 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:09.459040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:09.459022 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:43:18.323732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.323698 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-94c54b89b-7zxv2"] Nov 04 02:43:18.327461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.327440 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.383674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.383643 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-94c54b89b-7zxv2"] Nov 04 02:43:18.448493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.448457 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6k6g\" (UniqueName: \"kubernetes.io/projected/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-kube-api-access-s6k6g\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.448674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.448570 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-tls-cert\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.549951 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.549918 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6k6g\" (UniqueName: \"kubernetes.io/projected/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-kube-api-access-s6k6g\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.550141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.549980 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-tls-cert\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.552514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.552495 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-tls-cert\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.613434 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.613356 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6k6g\" (UniqueName: \"kubernetes.io/projected/100c05c7-eb70-4e04-93a4-e405b1f2c0fa-kube-api-access-s6k6g\") pod \"authorino-94c54b89b-7zxv2\" (UID: \"100c05c7-eb70-4e04-93a4-e405b1f2c0fa\") " pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:18.640558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:18.640528 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94c54b89b-7zxv2" Nov 04 02:43:19.013736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:19.013698 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-94c54b89b-7zxv2"] Nov 04 02:43:19.014739 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:43:19.014714 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod100c05c7_eb70_4e04_93a4_e405b1f2c0fa.slice/crio-d1a772790fcdfd400a24343ca6db5ee4aa17d917994a1a8375c17f3ae367145e WatchSource:0}: Error finding container d1a772790fcdfd400a24343ca6db5ee4aa17d917994a1a8375c17f3ae367145e: Status 404 returned error can't find the container with id d1a772790fcdfd400a24343ca6db5ee4aa17d917994a1a8375c17f3ae367145e Nov 04 02:43:19.530907 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:19.530866 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94c54b89b-7zxv2" event={"ID":"100c05c7-eb70-4e04-93a4-e405b1f2c0fa","Type":"ContainerStarted","Data":"d1a772790fcdfd400a24343ca6db5ee4aa17d917994a1a8375c17f3ae367145e"} Nov 04 02:43:21.544491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.544445 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94c54b89b-7zxv2" event={"ID":"100c05c7-eb70-4e04-93a4-e405b1f2c0fa","Type":"ContainerStarted","Data":"8f84d5b8127c019b931f7987fb35d35cbb8975ffbd9df1e59ff6781675b25602"} Nov 04 02:43:21.663214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.663148 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-94c54b89b-7zxv2" podStartSLOduration=2.015542753 podStartE2EDuration="3.66313484s" totalImagesPullingTime="1.647592087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:21.661702995 +0000 UTC m=+648.628204008" watchObservedRunningTime="2026-11-04 02:43:21.66313484 +0000 UTC m=+648.629635810" Nov 04 02:43:21.720237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.720200 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:43:21.720449 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.720419 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c64447b99-b8bpz" podUID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" containerName="authorino" containerID="cri-o://5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e" gracePeriod=30 Nov 04 02:43:21.978382 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.978355 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:43:21.984533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.984502 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\") pod \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " Nov 04 02:43:21.984671 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.984629 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\" (UniqueName: \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\") pod \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\" (UID: \"4c4d36e3-19f3-4117-bf5b-69e99e69398a\") " Nov 04 02:43:21.986911 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.986870 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7" pod "4c4d36e3-19f3-4117-bf5b-69e99e69398a" (UID: "4c4d36e3-19f3-4117-bf5b-69e99e69398a"). InnerVolumeSpecName "kube-api-access-w7dz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:43:21.997622 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:21.997590 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert" pod "4c4d36e3-19f3-4117-bf5b-69e99e69398a" (UID: "4c4d36e3-19f3-4117-bf5b-69e99e69398a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:43:22.085696 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.085660 4784 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4d36e3-19f3-4117-bf5b-69e99e69398a-tls-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:43:22.085845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.085703 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dz7\" (UniqueName: \"kubernetes.io/projected/4c4d36e3-19f3-4117-bf5b-69e99e69398a-kube-api-access-w7dz7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:43:22.551318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.551280 4784 generic.go:356] "Generic (PLEG): container finished" podID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" containerID="5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e" exitCode=0 Nov 04 02:43:22.551721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.551335 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c64447b99-b8bpz" Nov 04 02:43:22.551721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.551365 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c64447b99-b8bpz" event={"ID":"4c4d36e3-19f3-4117-bf5b-69e99e69398a","Type":"ContainerDied","Data":"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e"} Nov 04 02:43:22.551721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.551418 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c64447b99-b8bpz" event={"ID":"4c4d36e3-19f3-4117-bf5b-69e99e69398a","Type":"ContainerDied","Data":"04e974261668a1d3824bf217743fe32750739cf2d45501ebe71c1d4804843664"} Nov 04 02:43:22.551721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.551441 4784 scope.go:122] "RemoveContainer" containerID="5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e" Nov 04 02:43:22.562957 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.562935 4784 scope.go:122] "RemoveContainer" containerID="5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e" Nov 04 02:43:22.563326 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:43:22.563305 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e\": container with ID starting with 5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e not found: ID does not exist" containerID="5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e" Nov 04 02:43:22.563403 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.563340 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e"} err="failed to get container status \"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e\": rpc error: code = NotFound desc = could not find container \"5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e\": container with ID starting with 5499c52ee6a1ad5255e7ed195735b0b99f6955622e00bc668e0dfb97d29b246e not found: ID does not exist" Nov 04 02:43:22.723073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.723043 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:43:22.726710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:22.726685 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c64447b99-b8bpz"] Nov 04 02:43:23.161917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:23.161884 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" path="/var/lib/kubelet/pods/4c4d36e3-19f3-4117-bf5b-69e99e69398a/volumes" Nov 04 02:43:28.262512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.262478 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r82l2"] Nov 04 02:43:28.262908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.262882 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" containerName="authorino" Nov 04 02:43:28.262908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.262892 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" containerName="authorino" Nov 04 02:43:28.262979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.262973 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4d36e3-19f3-4117-bf5b-69e99e69398a" containerName="authorino" Nov 04 02:43:28.267534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.267516 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.342217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.342150 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgnt\" (UniqueName: \"kubernetes.io/projected/55946bcd-f633-4787-ac61-ce256b606eea-kube-api-access-csgnt\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.342416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.342242 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55946bcd-f633-4787-ac61-ce256b606eea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.371952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.371915 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9xsl8\"" Nov 04 02:43:28.372129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.371990 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 02:43:28.374136 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.374116 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r82l2"] Nov 04 02:43:28.429602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.429563 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bqh9n"] Nov 04 02:43:28.433168 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.433138 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.443636 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.443609 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5fb32-3e75-44b7-8e86-6498bc5af637-plugins-config-volume\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.443756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.443669 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgnt\" (UniqueName: \"kubernetes.io/projected/55946bcd-f633-4787-ac61-ce256b606eea-kube-api-access-csgnt\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.443815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.443769 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55946bcd-f633-4787-ac61-ce256b606eea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.443873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.443829 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw72w\" (UniqueName: \"kubernetes.io/projected/dad5fb32-3e75-44b7-8e86-6498bc5af637-kube-api-access-bw72w\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.444631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.444608 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55946bcd-f633-4787-ac61-ce256b606eea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.475041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.475007 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bqh9n"] Nov 04 02:43:28.507207 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.507159 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgnt\" (UniqueName: \"kubernetes.io/projected/55946bcd-f633-4787-ac61-ce256b606eea-kube-api-access-csgnt\") pod \"payload-pre-processing-7b98bb678c-r82l2\" (UID: \"55946bcd-f633-4787-ac61-ce256b606eea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.545376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.545282 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw72w\" (UniqueName: \"kubernetes.io/projected/dad5fb32-3e75-44b7-8e86-6498bc5af637-kube-api-access-bw72w\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.545542 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.545402 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5fb32-3e75-44b7-8e86-6498bc5af637-plugins-config-volume\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.546158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.546140 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5fb32-3e75-44b7-8e86-6498bc5af637-plugins-config-volume\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.580439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.580407 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:28.607861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.607836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw72w\" (UniqueName: \"kubernetes.io/projected/dad5fb32-3e75-44b7-8e86-6498bc5af637-kube-api-access-bw72w\") pod \"payload-processing-7bb9489676-bqh9n\" (UID: \"dad5fb32-3e75-44b7-8e86-6498bc5af637\") " pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.747045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.747000 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:28.964036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:28.964005 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r82l2"] Nov 04 02:43:28.965501 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:43:28.965475 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55946bcd_f633_4787_ac61_ce256b606eea.slice/crio-ce6c0e1b00c4a86de50386ffb93c66dfe7d6057fe5347354fb71451a3fa67e04 WatchSource:0}: Error finding container ce6c0e1b00c4a86de50386ffb93c66dfe7d6057fe5347354fb71451a3fa67e04: Status 404 returned error can't find the container with id ce6c0e1b00c4a86de50386ffb93c66dfe7d6057fe5347354fb71451a3fa67e04 Nov 04 02:43:29.123015 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:29.122985 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bqh9n"] Nov 04 02:43:29.124041 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:43:29.124015 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad5fb32_3e75_44b7_8e86_6498bc5af637.slice/crio-d409849eaa78ad1310d4c188a05c08f872e24fe01d3e482cf1d608695fa9e5a7 WatchSource:0}: Error finding container d409849eaa78ad1310d4c188a05c08f872e24fe01d3e482cf1d608695fa9e5a7: Status 404 returned error can't find the container with id d409849eaa78ad1310d4c188a05c08f872e24fe01d3e482cf1d608695fa9e5a7 Nov 04 02:43:29.599575 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:29.599526 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" event={"ID":"55946bcd-f633-4787-ac61-ce256b606eea","Type":"ContainerStarted","Data":"ce6c0e1b00c4a86de50386ffb93c66dfe7d6057fe5347354fb71451a3fa67e04"} Nov 04 02:43:29.601228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:29.601160 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" event={"ID":"dad5fb32-3e75-44b7-8e86-6498bc5af637","Type":"ContainerStarted","Data":"d409849eaa78ad1310d4c188a05c08f872e24fe01d3e482cf1d608695fa9e5a7"} Nov 04 02:43:32.649389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.649291 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" event={"ID":"55946bcd-f633-4787-ac61-ce256b606eea","Type":"ContainerStarted","Data":"68bd4c3eac5b755fae06029a2a3636179768c9e684677ded003bbbc5dca08db9"} Nov 04 02:43:32.649389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.649374 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:32.650947 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.650923 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" event={"ID":"dad5fb32-3e75-44b7-8e86-6498bc5af637","Type":"ContainerStarted","Data":"a558f89202b5ed3a794e39adc5a908e5eafd2cf3384ce30f5858440b35eeea4a"} Nov 04 02:43:32.651089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.651012 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:32.768173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.768114 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" podStartSLOduration=1.4359898260000001 podStartE2EDuration="4.768101231s" totalImagesPullingTime="3.332111405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:32.766996152 +0000 UTC m=+659.733497144" watchObservedRunningTime="2026-11-04 02:43:32.768101231 +0000 UTC m=+659.734602223" Nov 04 02:43:32.886759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:32.886706 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" podStartSLOduration=1.805543573 podStartE2EDuration="4.886690687s" totalImagesPullingTime="3.081147114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:32.884982746 +0000 UTC m=+659.851483737" watchObservedRunningTime="2026-11-04 02:43:32.886690687 +0000 UTC m=+659.853191679" Nov 04 02:43:35.626631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.626589 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4"] Nov 04 02:43:35.631556 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.631537 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.685169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.685138 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 02:43:35.736027 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.735990 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 02:43:35.736227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.736063 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9d55p\"" Nov 04 02:43:35.738472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.738451 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4"] Nov 04 02:43:35.774438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.774392 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c7a142-0ca3-4ed5-bfc5-7e75694180af-metrics-certs\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.774438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.774441 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhlv\" (UniqueName: \"kubernetes.io/projected/79c7a142-0ca3-4ed5-bfc5-7e75694180af-kube-api-access-4fhlv\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.876108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.876064 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c7a142-0ca3-4ed5-bfc5-7e75694180af-metrics-certs\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.876108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.876107 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhlv\" (UniqueName: \"kubernetes.io/projected/79c7a142-0ca3-4ed5-bfc5-7e75694180af-kube-api-access-4fhlv\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.878749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.878696 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c7a142-0ca3-4ed5-bfc5-7e75694180af-metrics-certs\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.938961 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.938927 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhlv\" (UniqueName: \"kubernetes.io/projected/79c7a142-0ca3-4ed5-bfc5-7e75694180af-kube-api-access-4fhlv\") pod \"ai-gateway-controller-8b7c7f7bc-xh4z4\" (UID: \"79c7a142-0ca3-4ed5-bfc5-7e75694180af\") " pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:35.944437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:35.944416 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:36.322520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:36.322488 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4"] Nov 04 02:43:36.323309 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:43:36.323279 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c7a142_0ca3_4ed5_bfc5_7e75694180af.slice/crio-907c60f834fd6285661241a2e789f56180b2bb0bfd1b212cbdfaaf3f03350676 WatchSource:0}: Error finding container 907c60f834fd6285661241a2e789f56180b2bb0bfd1b212cbdfaaf3f03350676: Status 404 returned error can't find the container with id 907c60f834fd6285661241a2e789f56180b2bb0bfd1b212cbdfaaf3f03350676 Nov 04 02:43:36.399225 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:43:36.399173 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:43:36.676998 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:36.676956 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" event={"ID":"79c7a142-0ca3-4ed5-bfc5-7e75694180af","Type":"ContainerStarted","Data":"907c60f834fd6285661241a2e789f56180b2bb0bfd1b212cbdfaaf3f03350676"} Nov 04 02:43:38.690455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:38.690415 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" event={"ID":"79c7a142-0ca3-4ed5-bfc5-7e75694180af","Type":"ContainerStarted","Data":"48b69ac12a74ebe13ef403e30dbd9023aeeef4344a6cd34ad561bd67e639a577"} Nov 04 02:43:38.690844 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:38.690476 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:43:38.898886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:38.898833 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" podStartSLOduration=1.944831471 podStartE2EDuration="3.898817204s" totalImagesPullingTime="1.953985733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:38.868573922 +0000 UTC m=+665.835074914" watchObservedRunningTime="2026-11-04 02:43:38.898817204 +0000 UTC m=+665.865318224" Nov 04 02:43:43.660098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:43.660023 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-bqh9n" Nov 04 02:43:43.660098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:43.660096 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r82l2" Nov 04 02:43:49.699703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:43:49.699674 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8b7c7f7bc-xh4z4" Nov 04 02:44:06.401054 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:44:06.401018 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:44:28.960091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:28.960055 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 02:44:28.963863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:28.963845 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.017659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.017625 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a0bcbc-openshift-default-dockercfg-z8ns2\"" Nov 04 02:44:29.072130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.072098 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 02:44:29.084895 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.084858 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.084909 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.084962 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085160 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085087 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085160 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085135 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085263 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085359 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqjv\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085497 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085474 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.085564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.085517 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186311 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186277 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186311 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186316 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186355 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186388 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqjv\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186421 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186437 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186481 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186504 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.186580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.186530 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.187046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.187019 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.187118 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.187048 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.187384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.187179 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.187384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.187230 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.187506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.187304 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.188980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.188958 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.189177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.189163 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.252250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.252143 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqjv\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.252457 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.252438 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.277410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.277381 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:29.593362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.593286 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 02:44:29.598016 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.597996 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.654704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.654671 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fe4e0c-openshift-default-dockercfg-gx9rt\"" Nov 04 02:44:29.663445 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.663410 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 02:44:29.692954 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.692899 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.692964 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5b2\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693041 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693105 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693150 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693173 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693237 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693334 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.693444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.693369 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.770942 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:29.761102 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0718ab_45bb_49ce_8d9f_216970d57496.slice/crio-2facf34394c49823d57a0a4e29c9f6e78c797c155117dfb9465baf8bcc602520 WatchSource:0}: Error finding container 2facf34394c49823d57a0a4e29c9f6e78c797c155117dfb9465baf8bcc602520: Status 404 returned error can't find the container with id 2facf34394c49823d57a0a4e29c9f6e78c797c155117dfb9465baf8bcc602520 Nov 04 02:44:29.770942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.764796 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:29.770942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.764896 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:29.770942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.764942 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:29.772566 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.772520 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 02:44:29.794112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794086 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794118 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794244 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794278 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5b2\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794312 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794350 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794376 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794395 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794664 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794432 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.794664 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794647 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.795257 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.794966 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.795257 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.795221 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.795483 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.795442 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.795575 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.795513 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.796787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.796769 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.798154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.798130 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.872024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.871945 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.872024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.872005 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5b2\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.912108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.912070 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:29.993269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.992358 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerStarted","Data":"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972"} Nov 04 02:44:29.993269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:29.992423 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerStarted","Data":"2facf34394c49823d57a0a4e29c9f6e78c797c155117dfb9465baf8bcc602520"} Nov 04 02:44:30.117610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.117548 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podStartSLOduration=2.1175270409999998 podStartE2EDuration="2.117527041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:30.115855818 +0000 UTC m=+717.082356812" watchObservedRunningTime="2026-11-04 02:44:30.117527041 +0000 UTC m=+717.084028035" Nov 04 02:44:30.277746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.277705 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:30.279812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.279783 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 02:44:30.284800 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.284778 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:30.284931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.284856 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.344994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.344926 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-203c35-openshift-default-dockercfg-ggkdg\"" Nov 04 02:44:30.345864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.345839 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 02:44:30.350249 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:30.350218 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96afbb02_6497_4a26_9975_f0340b4e0bd1.slice/crio-2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c WatchSource:0}: Error finding container 2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c: Status 404 returned error can't find the container with id 2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c Nov 04 02:44:30.352752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.352709 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:30.352835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.352784 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:30.352835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.352823 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:30.395725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.395698 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 02:44:30.404687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.404662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.404892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.404865 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.404924 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405079 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.404986 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.405080 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.405133 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.405171 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.405235 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlmc\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.405405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.405296 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506591 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506685 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506720 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506764 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506787 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506813 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506849 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506887 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.506938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506933 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlmc\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.507058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.506964 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.507758 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.507662 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.507758 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.507740 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.508129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.507933 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.508129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.507993 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.509452 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.509420 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.509663 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.509644 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.572502 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.572428 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.572647 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.572604 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlmc\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.614023 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.613988 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:30.921783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.921745 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 02:44:30.927075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.927053 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:30.988384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:30.988348 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 02:44:31.001707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.001669 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerStarted","Data":"9c81c4ed1d9b44d88fcb04b53736f93080a1e71c3893e9f2138737ea3ff28730"} Nov 04 02:44:31.002130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.001738 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerStarted","Data":"2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c"} Nov 04 02:44:31.002130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.002026 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:31.003364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.003341 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:44:31.005432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.005400 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ebb7fd-openshift-default-dockercfg-k9smx\"" Nov 04 02:44:31.050798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.050768 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 02:44:31.051636 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:31.051603 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b0097c_5696_43a3_a6cd_74ff8d85897f.slice/crio-af60dbd489457aa43fcda35129f04e49a129f8fd5c53a3c99eca7d7b394d5803 WatchSource:0}: Error finding container af60dbd489457aa43fcda35129f04e49a129f8fd5c53a3c99eca7d7b394d5803: Status 404 returned error can't find the container with id af60dbd489457aa43fcda35129f04e49a129f8fd5c53a3c99eca7d7b394d5803 Nov 04 02:44:31.054040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.054003 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:31.054143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.054089 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:31.054143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.054136 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:31.112753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.112727 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.112896 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.112765 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.112896 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.112793 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.112896 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.112849 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrzk\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.113590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.113042 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.113590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.113242 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.113590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.113308 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.113590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.113345 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.113590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.113374 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215829 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215886 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215909 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215934 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215970 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.215994 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.216016 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.216035 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrzk\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.216392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.216068 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.217173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.216509 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.220223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.220114 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.220436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.220414 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.221426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.221325 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.221726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.221560 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.231876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.231808 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.248220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.242365 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.284340 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.284307 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.284567 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.284432 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrzk\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.302928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.302865 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podStartSLOduration=2.302846037 podStartE2EDuration="2.302846037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:31.301654438 +0000 UTC m=+718.268155431" watchObservedRunningTime="2026-11-04 02:44:31.302846037 +0000 UTC m=+718.269347029" Nov 04 02:44:31.540401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.540318 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:31.913058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.912980 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:31.918268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.918240 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:31.924153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.924125 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 02:44:31.924583 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:31.924540 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa1207c_4bb6_47e1_ae6d_4398909b19c7.slice/crio-c9717b53faeee5b289bb7f93d20d96f3d6904ca9d5653b2217c8708083f1ec61 WatchSource:0}: Error finding container c9717b53faeee5b289bb7f93d20d96f3d6904ca9d5653b2217c8708083f1ec61: Status 404 returned error can't find the container with id c9717b53faeee5b289bb7f93d20d96f3d6904ca9d5653b2217c8708083f1ec61 Nov 04 02:44:31.926941 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.926908 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:31.927019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.926975 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:31.927019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:31.927003 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:44:32.010663 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.010637 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerStarted","Data":"cdb281b33af22ea6baecae9164d34defff666dc1bd176aeff6a2cf7c0688268d"} Nov 04 02:44:32.011077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.010700 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerStarted","Data":"af60dbd489457aa43fcda35129f04e49a129f8fd5c53a3c99eca7d7b394d5803"} Nov 04 02:44:32.012432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.012408 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerStarted","Data":"c9717b53faeee5b289bb7f93d20d96f3d6904ca9d5653b2217c8708083f1ec61"} Nov 04 02:44:32.012628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.012612 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:32.013607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.013590 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:44:32.228318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.228263 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podStartSLOduration=2.22824918 podStartE2EDuration="2.22824918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:32.224897789 +0000 UTC m=+719.191398799" watchObservedRunningTime="2026-11-04 02:44:32.22824918 +0000 UTC m=+719.194750172" Nov 04 02:44:32.473374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.473323 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podStartSLOduration=2.473305172 podStartE2EDuration="2.473305172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:32.472385605 +0000 UTC m=+719.438886596" watchObservedRunningTime="2026-11-04 02:44:32.473305172 +0000 UTC m=+719.439806222" Nov 04 02:44:32.540749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.540659 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:32.614863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.614820 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:32.620484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:32.620450 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:33.019133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:33.019097 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerStarted","Data":"99174578f0e7cde51294e91604875b1a1c65e2c63d5dc0b8cf3641bc618f159a"} Nov 04 02:44:33.019711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:33.019696 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:33.020455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:33.020436 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:44:33.548288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:33.548253 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:34.025343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:34.025309 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:34.026257 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:34.026238 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:44:36.400904 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:44:36.400866 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:44:38.022035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.022000 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 02:44:38.027281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.027255 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.083323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.083294 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 02:44:38.096829 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.096797 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.096999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.096845 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.130795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.130759 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a0bcbc-dockercfg-4rk4g\"" Nov 04 02:44:38.130795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.130765 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a0bcbc\"" Nov 04 02:44:38.198466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.198427 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.198466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.198470 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.199218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.199170 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.260907 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.260874 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\") pod \"payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.340460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.340367 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:38.723714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:38.723682 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 02:44:38.724976 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:38.724951 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e41c29_9039_4264_aee1_3e6b7d27444a.slice/crio-7d9c136fdc23151faf2608177474173d9c7b1475c536bb426ec6446fc9693312 WatchSource:0}: Error finding container 7d9c136fdc23151faf2608177474173d9c7b1475c536bb426ec6446fc9693312: Status 404 returned error can't find the container with id 7d9c136fdc23151faf2608177474173d9c7b1475c536bb426ec6446fc9693312 Nov 04 02:44:39.059806 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:39.059415 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" event={"ID":"88e41c29-9039-4264-aee1-3e6b7d27444a","Type":"ContainerStarted","Data":"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c"} Nov 04 02:44:39.059806 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:39.059494 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" event={"ID":"88e41c29-9039-4264-aee1-3e6b7d27444a","Type":"ContainerStarted","Data":"7d9c136fdc23151faf2608177474173d9c7b1475c536bb426ec6446fc9693312"} Nov 04 02:44:39.059806 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:39.059521 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:39.186939 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:39.186884 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" podStartSLOduration=2.186868936 podStartE2EDuration="2.186868936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:39.185736068 +0000 UTC m=+726.152237060" watchObservedRunningTime="2026-11-04 02:44:39.186868936 +0000 UTC m=+726.153369928" Nov 04 02:44:42.897086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:42.897050 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 02:44:42.900703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:42.900686 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:42.950672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:42.950638 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbhf\" (UniqueName: \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:42.950878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:42.950804 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:42.954908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:42.954877 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-cdcf67\"" Nov 04 02:44:43.016170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.016135 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-cdcf67-dockercfg-kklkb\"" Nov 04 02:44:43.028431 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.024917 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 02:44:43.051736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.051696 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbhf\" (UniqueName: \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:43.051928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.051821 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:43.052825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.052795 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:43.114453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.114419 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbhf\" (UniqueName: \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\") pod \"payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:43.214220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.214145 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:43.590109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:43.590080 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 02:44:43.591066 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:43.591039 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea2964d_1f82_40ca_855f_211bf8b60f0e.slice/crio-ca95f639f251bc6257df8255d63ca9abfee95a99cd3ff11ff4a991aa74c58c3c WatchSource:0}: Error finding container ca95f639f251bc6257df8255d63ca9abfee95a99cd3ff11ff4a991aa74c58c3c: Status 404 returned error can't find the container with id ca95f639f251bc6257df8255d63ca9abfee95a99cd3ff11ff4a991aa74c58c3c Nov 04 02:44:44.090452 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:44.090416 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" event={"ID":"4ea2964d-1f82-40ca-855f-211bf8b60f0e","Type":"ContainerStarted","Data":"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281"} Nov 04 02:44:44.090967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:44.090479 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" event={"ID":"4ea2964d-1f82-40ca-855f-211bf8b60f0e","Type":"ContainerStarted","Data":"ca95f639f251bc6257df8255d63ca9abfee95a99cd3ff11ff4a991aa74c58c3c"} Nov 04 02:44:44.090967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:44.090507 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:44.209944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:44.209887 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" podStartSLOduration=2.20987344 podStartE2EDuration="2.20987344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:44.208332577 +0000 UTC m=+731.174833580" watchObservedRunningTime="2026-11-04 02:44:44.20987344 +0000 UTC m=+731.176374428" Nov 04 02:44:47.791017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.790972 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 02:44:47.796539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.796511 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:47.850819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.850787 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 02:44:47.898453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.898421 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-fe4e0c-dockercfg-rjf74\"" Nov 04 02:44:47.898620 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.898460 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fe4e0c\"" Nov 04 02:44:47.902030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.901999 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4qx\" (UniqueName: \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:47.902174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:47.902061 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.003214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.003086 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4qx\" (UniqueName: \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.003214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.003173 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.004251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.004217 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.066534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.066451 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4qx\" (UniqueName: \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\") pod \"payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.110627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.110590 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:48.516280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:48.516242 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 02:44:48.517023 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:48.516997 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad192d2_12d2_4da2_a496_a632d91889fc.slice/crio-ad0d5e6353de747309642f2846eb02bd333396ceffa0e1a703b782b671874280 WatchSource:0}: Error finding container ad0d5e6353de747309642f2846eb02bd333396ceffa0e1a703b782b671874280: Status 404 returned error can't find the container with id ad0d5e6353de747309642f2846eb02bd333396ceffa0e1a703b782b671874280 Nov 04 02:44:49.121281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:49.121245 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" event={"ID":"9ad192d2-12d2-4da2-a496-a632d91889fc","Type":"ContainerStarted","Data":"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50"} Nov 04 02:44:49.121699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:49.121302 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" event={"ID":"9ad192d2-12d2-4da2-a496-a632d91889fc","Type":"ContainerStarted","Data":"ad0d5e6353de747309642f2846eb02bd333396ceffa0e1a703b782b671874280"} Nov 04 02:44:49.121699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:49.121326 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:44:49.242108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:49.242052 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" podStartSLOduration=2.242036895 podStartE2EDuration="2.242036895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:49.239377123 +0000 UTC m=+736.205878109" watchObservedRunningTime="2026-11-04 02:44:49.242036895 +0000 UTC m=+736.208537886" Nov 04 02:44:50.069656 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.069626 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 02:44:50.292980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.292949 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 02:44:50.296880 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.296860 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.334688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.334598 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.334688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.334663 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsq4\" (UniqueName: \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.350460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.350429 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3b7476-dockercfg-4h9db\"" Nov 04 02:44:50.350676 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.350492 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3b7476\"" Nov 04 02:44:50.354653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.354625 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 02:44:50.435926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.435892 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.436117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.435956 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsq4\" (UniqueName: \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.436653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.436634 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.499114 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.499074 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsq4\" (UniqueName: \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\") pod \"payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.610229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.610130 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:50.993230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:50.993202 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 02:44:50.994806 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:50.994776 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9246b627_cbd1_4314_9e93_f3b843db317d.slice/crio-d9d6d81c5d16bfb995180bbfee14ca62ed7896727d86ff2e4f756b66cadcaff0 WatchSource:0}: Error finding container d9d6d81c5d16bfb995180bbfee14ca62ed7896727d86ff2e4f756b66cadcaff0: Status 404 returned error can't find the container with id d9d6d81c5d16bfb995180bbfee14ca62ed7896727d86ff2e4f756b66cadcaff0 Nov 04 02:44:51.136997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:51.136960 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" event={"ID":"9246b627-cbd1-4314-9e93-f3b843db317d","Type":"ContainerStarted","Data":"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec"} Nov 04 02:44:51.137236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:51.137012 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" event={"ID":"9246b627-cbd1-4314-9e93-f3b843db317d","Type":"ContainerStarted","Data":"d9d6d81c5d16bfb995180bbfee14ca62ed7896727d86ff2e4f756b66cadcaff0"} Nov 04 02:44:51.137236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:51.137113 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:44:51.258947 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:51.258848 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" podStartSLOduration=1.2588347899999999 podStartE2EDuration="1.25883479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:51.256630065 +0000 UTC m=+738.223131057" watchObservedRunningTime="2026-11-04 02:44:51.25883479 +0000 UTC m=+738.225335781" Nov 04 02:44:52.837551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.837514 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 02:44:52.842432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.842409 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:52.898686 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.898646 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 02:44:52.949912 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.949871 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e516ff-dockercfg-kmmpc\"" Nov 04 02:44:52.950122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.950021 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e516ff\"" Nov 04 02:44:52.966564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.966366 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzh2z\" (UniqueName: \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:52.966564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:52.966471 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.003728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.003690 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 02:44:53.007645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.007625 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.054690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.054652 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 02:44:53.068016 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.067980 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzh2z\" (UniqueName: \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.068237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.068103 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.068237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.068181 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdg6\" (UniqueName: \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.068371 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.068300 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.068807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.068788 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.133463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.133394 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzh2z\" (UniqueName: \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\") pod \"payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.155892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.155861 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:53.169758 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.169727 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdg6\" (UniqueName: \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.169887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.169813 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.170490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.170467 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.233174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.233135 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdg6\" (UniqueName: \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\") pod \"payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.320581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.320539 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:53.533062 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.533030 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 02:44:53.533760 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:53.533723 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521cf223_72d7_45cf_b3a6_30871e1a5765.slice/crio-32a4e8ed2a433ed38e3b74c9ed96c58381350578dc6ae47f7adf1eb76a9afe17 WatchSource:0}: Error finding container 32a4e8ed2a433ed38e3b74c9ed96c58381350578dc6ae47f7adf1eb76a9afe17: Status 404 returned error can't find the container with id 32a4e8ed2a433ed38e3b74c9ed96c58381350578dc6ae47f7adf1eb76a9afe17 Nov 04 02:44:53.701782 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:53.701753 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 02:44:53.701936 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:53.701913 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66543109_7ff8_4a52_b49a_cf9712812777.slice/crio-90c9c129799e411bce2ec3c6081b9ed13fe1cd1476e88b2831ddac42322c201a WatchSource:0}: Error finding container 90c9c129799e411bce2ec3c6081b9ed13fe1cd1476e88b2831ddac42322c201a: Status 404 returned error can't find the container with id 90c9c129799e411bce2ec3c6081b9ed13fe1cd1476e88b2831ddac42322c201a Nov 04 02:44:54.156455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.156422 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" event={"ID":"66543109-7ff8-4a52-b49a-cf9712812777","Type":"ContainerStarted","Data":"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b"} Nov 04 02:44:54.156926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.156467 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" event={"ID":"66543109-7ff8-4a52-b49a-cf9712812777","Type":"ContainerStarted","Data":"90c9c129799e411bce2ec3c6081b9ed13fe1cd1476e88b2831ddac42322c201a"} Nov 04 02:44:54.156926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.156491 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:44:54.158228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.158179 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" event={"ID":"521cf223-72d7-45cf-b3a6-30871e1a5765","Type":"ContainerStarted","Data":"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175"} Nov 04 02:44:54.158365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.158241 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" event={"ID":"521cf223-72d7-45cf-b3a6-30871e1a5765","Type":"ContainerStarted","Data":"32a4e8ed2a433ed38e3b74c9ed96c58381350578dc6ae47f7adf1eb76a9afe17"} Nov 04 02:44:54.158365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.158261 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:44:54.276552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.276496 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" podStartSLOduration=2.27648153 podStartE2EDuration="2.27648153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:54.274690655 +0000 UTC m=+741.241191644" watchObservedRunningTime="2026-11-04 02:44:54.27648153 +0000 UTC m=+741.242982522" Nov 04 02:44:54.395817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:54.395755 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" podStartSLOduration=2.395737998 podStartE2EDuration="2.395737998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:54.392852307 +0000 UTC m=+741.359353301" watchObservedRunningTime="2026-11-04 02:44:54.395737998 +0000 UTC m=+741.362239015" Nov 04 02:44:55.099931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.099901 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 02:44:55.352364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.352280 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 02:44:55.356453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.356427 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.397623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.397587 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qhq\" (UniqueName: \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.397816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.397699 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.410428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.410384 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-203c35\"" Nov 04 02:44:55.410581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.410402 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-203c35-dockercfg-6k65x\"" Nov 04 02:44:55.413654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.413627 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 02:44:55.521206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.521140 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qhq\" (UniqueName: \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.521427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.521262 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.523242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.522633 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.525799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.525772 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 02:44:55.534703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.534661 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.566826 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.566791 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 02:44:55.584821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.584785 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qhq\" (UniqueName: \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\") pod \"payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.622515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.622426 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.622515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.622515 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88sk\" (UniqueName: \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.675430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.675398 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:55.724396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.724355 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.724629 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.724468 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88sk\" (UniqueName: \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.725657 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.725631 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.797433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.796720 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88sk\" (UniqueName: \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\") pod \"payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:55.851072 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:55.851039 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:56.062627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.062594 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 02:44:56.062806 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:56.062782 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab74e6d_4c65_42aa_99d1_42e8146a943f.slice/crio-ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d WatchSource:0}: Error finding container ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d: Status 404 returned error can't find the container with id ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d Nov 04 02:44:56.178958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.178864 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" event={"ID":"7ab74e6d-4c65-42aa-99d1-42e8146a943f","Type":"ContainerStarted","Data":"0b50b0474742bfab43528e56f5f76dbd304ac021aba120c03c015e85898646e7"} Nov 04 02:44:56.178958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.178926 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" event={"ID":"7ab74e6d-4c65-42aa-99d1-42e8146a943f","Type":"ContainerStarted","Data":"ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d"} Nov 04 02:44:56.178958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.178955 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:44:56.242243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.242216 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 02:44:56.244109 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:44:56.244080 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02459976_aea1_488d_83f7_f2525df146ec.slice/crio-d8fc1172616130d02a809e5e23585fb4f7219fae5d300c614d2979c41ae0ab45 WatchSource:0}: Error finding container d8fc1172616130d02a809e5e23585fb4f7219fae5d300c614d2979c41ae0ab45: Status 404 returned error can't find the container with id d8fc1172616130d02a809e5e23585fb4f7219fae5d300c614d2979c41ae0ab45 Nov 04 02:44:56.299785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:56.299741 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" podStartSLOduration=1.299726789 podStartE2EDuration="1.299726789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:56.297796378 +0000 UTC m=+743.264297370" watchObservedRunningTime="2026-11-04 02:44:56.299726789 +0000 UTC m=+743.266227780" Nov 04 02:44:57.187849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:57.187804 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" event={"ID":"02459976-aea1-488d-83f7-f2525df146ec","Type":"ContainerStarted","Data":"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad"} Nov 04 02:44:57.188292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:57.187874 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" event={"ID":"02459976-aea1-488d-83f7-f2525df146ec","Type":"ContainerStarted","Data":"d8fc1172616130d02a809e5e23585fb4f7219fae5d300c614d2979c41ae0ab45"} Nov 04 02:44:57.188292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:57.187907 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:44:57.379216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:44:57.360039 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" podStartSLOduration=2.3600202599999998 podStartE2EDuration="2.36002026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:57.305793794 +0000 UTC m=+744.272294813" watchObservedRunningTime="2026-11-04 02:44:57.36002026 +0000 UTC m=+744.326521253" Nov 04 02:45:00.131348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.131319 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 02:45:00.200216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.199655 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:00.212234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.212205 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.212746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.212711 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:00.217289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.217267 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.239682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.239652 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:00.284248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.279538 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:45:00.284248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.279655 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:45:00.284762 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.284721 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:45:00.290270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.290094 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:00.309674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.309629 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbl2\" (UniqueName: \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.309873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.309688 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.309939 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.309865 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h447\" (UniqueName: \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.309997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.309978 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.411892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.411800 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbl2\" (UniqueName: \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.411892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.411853 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.412122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.411939 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h447\" (UniqueName: \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.412122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.411988 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.412773 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.412746 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.414089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.414062 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.487917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.487883 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbl2\" (UniqueName: \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.490078 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.490048 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h447\" (UniqueName: \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:00.529612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.529574 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:00.536484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:00.536275 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:01.051434 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:01.051395 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:01.052268 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:45:01.052243 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283d2e6a_22d3_4aa9_95d5_d5069aa2024a.slice/crio-a22dbcc187d4e6815eae590d8f82413514368b6163a3d25261de91329c0a7278 WatchSource:0}: Error finding container a22dbcc187d4e6815eae590d8f82413514368b6163a3d25261de91329c0a7278: Status 404 returned error can't find the container with id a22dbcc187d4e6815eae590d8f82413514368b6163a3d25261de91329c0a7278 Nov 04 02:45:01.088607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:01.088578 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:01.223084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:01.223049 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerStarted","Data":"a22dbcc187d4e6815eae590d8f82413514368b6163a3d25261de91329c0a7278"} Nov 04 02:45:01.224453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:01.224430 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerStarted","Data":"70bfcbf9775882419d812722ea404fc86ccfb81129c27fe2d7d85b19a38de29a"} Nov 04 02:45:02.146282 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:02.146254 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 02:45:05.216138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:05.216112 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 02:45:05.216690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:05.216170 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 02:45:06.424087 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:06.424040 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:45:07.191667 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.191642 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 02:45:07.310974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.310929 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerStarted","Data":"c0b0f625bf5d747290d500d9fcd1afd6d550deecca8a82ca4ce210f310d9c5f2"} Nov 04 02:45:07.312878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.312851 4784 generic.go:356] "Generic (PLEG): container finished" podID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerID="9c81c4ed1d9b44d88fcb04b53736f93080a1e71c3893e9f2138737ea3ff28730" exitCode=0 Nov 04 02:45:07.313032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.312935 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerDied","Data":"9c81c4ed1d9b44d88fcb04b53736f93080a1e71c3893e9f2138737ea3ff28730"} Nov 04 02:45:07.313330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.313311 4784 scope.go:122] "RemoveContainer" containerID="9c81c4ed1d9b44d88fcb04b53736f93080a1e71c3893e9f2138737ea3ff28730" Nov 04 02:45:07.315643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.314535 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:07.315643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.314596 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:07.315643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.314630 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:07.439328 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:07.439267 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" podStartSLOduration=1.7222374029999998 podStartE2EDuration="7.439243519s" totalImagesPullingTime="5.717006116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:07.438458561 +0000 UTC m=+754.404959553" watchObservedRunningTime="2026-11-04 02:45:07.439243519 +0000 UTC m=+754.405744513" Nov 04 02:45:08.209308 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.208319 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 02:45:08.320989 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.320909 4784 generic.go:356] "Generic (PLEG): container finished" podID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerID="060e64330f5494b48590b77e28ccd49180d1836420be937aec4b256c300d79ac" exitCode=60 Nov 04 02:45:08.321241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.320995 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerDied","Data":"060e64330f5494b48590b77e28ccd49180d1836420be937aec4b256c300d79ac"} Nov 04 02:45:08.321337 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.321316 4784 scope.go:122] "RemoveContainer" containerID="060e64330f5494b48590b77e28ccd49180d1836420be937aec4b256c300d79ac" Nov 04 02:45:08.325574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.325538 4784 generic.go:356] "Generic (PLEG): container finished" podID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerID="c0b0f625bf5d747290d500d9fcd1afd6d550deecca8a82ca4ce210f310d9c5f2" exitCode=60 Nov 04 02:45:08.325756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.325676 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerDied","Data":"c0b0f625bf5d747290d500d9fcd1afd6d550deecca8a82ca4ce210f310d9c5f2"} Nov 04 02:45:08.326050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.325948 4784 scope.go:122] "RemoveContainer" containerID="c0b0f625bf5d747290d500d9fcd1afd6d550deecca8a82ca4ce210f310d9c5f2" Nov 04 02:45:08.328089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.328067 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerStarted","Data":"06b157aa881f72e9ee55f0c012315e0537546b8cd87f066ee1ac6a66f57d2666"} Nov 04 02:45:08.913266 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.913039 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:45:08.913704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.913369 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:45:08.913704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:08.913415 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 02:45:09.335574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:09.335533 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerStarted","Data":"cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1"} Nov 04 02:45:09.339599 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:09.339572 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerStarted","Data":"4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6"} Nov 04 02:45:09.491091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:09.491019 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" podStartSLOduration=3.605045466 podStartE2EDuration="9.490998318s" totalImagesPullingTime="5.885952852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:09.472595422 +0000 UTC m=+756.439096416" watchObservedRunningTime="2026-11-04 02:45:09.490998318 +0000 UTC m=+756.457499311" Nov 04 02:45:09.913251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:09.913120 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:45:09.913251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:09.913181 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 02:45:10.346479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.346441 4784 generic.go:356] "Generic (PLEG): container finished" podID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerID="cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1" exitCode=60 Nov 04 02:45:10.346691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.346524 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerDied","Data":"cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1"} Nov 04 02:45:10.346691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.346587 4784 scope.go:122] "RemoveContainer" containerID="060e64330f5494b48590b77e28ccd49180d1836420be937aec4b256c300d79ac" Nov 04 02:45:10.346871 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.346853 4784 scope.go:122] "RemoveContainer" containerID="cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1" Nov 04 02:45:10.347160 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:10.347128 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26_odh-ai-gateway-infra(283d2e6a-22d3-4aa9-95d5-d5069aa2024a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" Nov 04 02:45:10.348752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.348732 4784 generic.go:356] "Generic (PLEG): container finished" podID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerID="4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6" exitCode=60 Nov 04 02:45:10.348836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.348813 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerDied","Data":"4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6"} Nov 04 02:45:10.349045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.349030 4784 scope.go:122] "RemoveContainer" containerID="4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6" Nov 04 02:45:10.349326 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:10.349309 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk_odh-ai-gateway-infra(ae5f1f06-ca99-4bbd-a290-b466927b15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" Nov 04 02:45:10.389468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.389445 4784 scope.go:122] "RemoveContainer" containerID="c0b0f625bf5d747290d500d9fcd1afd6d550deecca8a82ca4ce210f310d9c5f2" Nov 04 02:45:10.912799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.912759 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:45:10.912993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:10.912830 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 02:45:11.389215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:11.388626 4784 scope.go:122] "RemoveContainer" containerID="cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1" Nov 04 02:45:11.389215 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:11.388931 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26_odh-ai-gateway-infra(283d2e6a-22d3-4aa9-95d5-d5069aa2024a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" Nov 04 02:45:11.391974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:11.391697 4784 scope.go:122] "RemoveContainer" containerID="4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6" Nov 04 02:45:11.391974 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:11.391927 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk_odh-ai-gateway-infra(ae5f1f06-ca99-4bbd-a290-b466927b15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" Nov 04 02:45:11.912758 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:11.912724 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:45:11.913002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:11.912976 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 02:45:12.912635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:12.912596 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:45:12.913052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:12.912656 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 02:45:13.918043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:13.918012 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:45:13.919548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:13.918355 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:45:13.919548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:13.919240 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 02:45:14.414799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.414769 4784 generic.go:356] "Generic (PLEG): container finished" podID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerID="136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972" exitCode=0 Nov 04 02:45:14.415042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.414843 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerDied","Data":"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972"} Nov 04 02:45:14.415619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.415419 4784 scope.go:122] "RemoveContainer" containerID="136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972" Nov 04 02:45:14.417125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.417084 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:14.417627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.417246 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:14.417627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:14.417305 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:15.424179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:15.424144 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerStarted","Data":"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd"} Nov 04 02:45:16.001973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:16.001945 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:45:16.280465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:16.280372 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:45:16.282150 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:16.282123 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:45:16.282323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:16.282181 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 02:45:17.280976 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:17.280610 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:45:17.280976 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:17.280662 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 02:45:18.280698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:18.280654 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:45:18.280886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:18.280717 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 02:45:19.281334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:19.281296 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:45:19.281965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:19.281931 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 02:45:20.280665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:20.280555 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:45:20.280665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:20.280631 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 02:45:21.283165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:21.283129 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:45:21.284333 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:21.284309 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 02:45:22.153151 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:22.153111 4784 scope.go:122] "RemoveContainer" containerID="cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1" Nov 04 02:45:23.164357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:23.164323 4784 scope.go:122] "RemoveContainer" containerID="4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6" Nov 04 02:45:23.476640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:23.476604 4784 generic.go:356] "Generic (PLEG): container finished" podID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerID="e47614769d1d90f2b438328861636300ae3ee1c6f1fa630cee9daa0179b64c3c" exitCode=60 Nov 04 02:45:23.476791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:23.476684 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerDied","Data":"e47614769d1d90f2b438328861636300ae3ee1c6f1fa630cee9daa0179b64c3c"} Nov 04 02:45:23.476791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:23.476755 4784 scope.go:122] "RemoveContainer" containerID="cc14114e6e32455d7f1398fff43da40c750020aef3152c7410083974608ad0e1" Nov 04 02:45:23.477162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:23.477137 4784 scope.go:122] "RemoveContainer" containerID="e47614769d1d90f2b438328861636300ae3ee1c6f1fa630cee9daa0179b64c3c" Nov 04 02:45:23.477500 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:23.477470 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26_odh-ai-gateway-infra(283d2e6a-22d3-4aa9-95d5-d5069aa2024a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" Nov 04 02:45:24.488464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.488430 4784 generic.go:356] "Generic (PLEG): container finished" podID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerID="59e2e11fe0cba7c94ec0b2080a43c9c0ef7c4310513e46a51771c7bad9df0812" exitCode=60 Nov 04 02:45:24.488913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.488498 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerDied","Data":"59e2e11fe0cba7c94ec0b2080a43c9c0ef7c4310513e46a51771c7bad9df0812"} Nov 04 02:45:24.488913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.488558 4784 scope.go:122] "RemoveContainer" containerID="4355b8fe3fda4aebb26be05f349e1cd5245c9772c1edd37ee080e51c8747f7f6" Nov 04 02:45:24.488913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.488878 4784 scope.go:122] "RemoveContainer" containerID="59e2e11fe0cba7c94ec0b2080a43c9c0ef7c4310513e46a51771c7bad9df0812" Nov 04 02:45:24.489181 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:24.489161 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk_odh-ai-gateway-infra(ae5f1f06-ca99-4bbd-a290-b466927b15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" Nov 04 02:45:24.610233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.610167 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:24.769679 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.769647 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:24.814498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.814458 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\" (UniqueName: \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\") pod \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " Nov 04 02:45:24.814498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.814498 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\") pod \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\" (UID: \"283d2e6a-22d3-4aa9-95d5-d5069aa2024a\") " Nov 04 02:45:24.814894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.814864 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca" pod "283d2e6a-22d3-4aa9-95d5-d5069aa2024a" (UID: "283d2e6a-22d3-4aa9-95d5-d5069aa2024a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:45:24.816808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.816785 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447" pod "283d2e6a-22d3-4aa9-95d5-d5069aa2024a" (UID: "283d2e6a-22d3-4aa9-95d5-d5069aa2024a"). InnerVolumeSpecName "kube-api-access-8h447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:45:24.915380 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.915350 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h447\" (UniqueName: \"kubernetes.io/projected/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-kube-api-access-8h447\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:24.915380 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:24.915376 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/283d2e6a-22d3-4aa9-95d5-d5069aa2024a-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:25.495876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.495790 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" Nov 04 02:45:25.495876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.495788 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26" event={"ID":"283d2e6a-22d3-4aa9-95d5-d5069aa2024a","Type":"ContainerDied","Data":"a22dbcc187d4e6815eae590d8f82413514368b6163a3d25261de91329c0a7278"} Nov 04 02:45:25.496454 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.495921 4784 scope.go:122] "RemoveContainer" containerID="e47614769d1d90f2b438328861636300ae3ee1c6f1fa630cee9daa0179b64c3c" Nov 04 02:45:25.621248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.621214 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:25.672132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.672094 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:25.677902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.677872 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896005-4nk26"] Nov 04 02:45:25.802973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.802945 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:25.934318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.934283 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\") pod \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " Nov 04 02:45:25.934508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.934463 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\" (UniqueName: \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\") pod \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\" (UID: \"ae5f1f06-ca99-4bbd-a290-b466927b15a4\") " Nov 04 02:45:25.934733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.934705 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca" pod "ae5f1f06-ca99-4bbd-a290-b466927b15a4" (UID: "ae5f1f06-ca99-4bbd-a290-b466927b15a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:45:25.934934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.934907 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f1f06-ca99-4bbd-a290-b466927b15a4-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:25.936769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:25.936745 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2" pod "ae5f1f06-ca99-4bbd-a290-b466927b15a4" (UID: "ae5f1f06-ca99-4bbd-a290-b466927b15a4"). InnerVolumeSpecName "kube-api-access-2cbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:45:26.035857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.035766 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbl2\" (UniqueName: \"kubernetes.io/projected/ae5f1f06-ca99-4bbd-a290-b466927b15a4-kube-api-access-2cbl2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:26.506340 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.506302 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" Nov 04 02:45:26.506827 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.506338 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk" event={"ID":"ae5f1f06-ca99-4bbd-a290-b466927b15a4","Type":"ContainerDied","Data":"70bfcbf9775882419d812722ea404fc86ccfb81129c27fe2d7d85b19a38de29a"} Nov 04 02:45:26.506827 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.506417 4784 scope.go:122] "RemoveContainer" containerID="59e2e11fe0cba7c94ec0b2080a43c9c0ef7c4310513e46a51771c7bad9df0812" Nov 04 02:45:26.676548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.676504 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:26.680296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:26.680268 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896005-89fxk"] Nov 04 02:45:27.160322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:27.160284 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" path="/var/lib/kubelet/pods/283d2e6a-22d3-4aa9-95d5-d5069aa2024a/volumes" Nov 04 02:45:27.160704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:27.160690 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" path="/var/lib/kubelet/pods/ae5f1f06-ca99-4bbd-a290-b466927b15a4/volumes" Nov 04 02:45:36.422398 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:45:36.422355 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:45:47.120833 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.120727 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 02:45:47.121559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121536 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121561 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121577 4784 cpu_manager.go:414] "containerMap: removing container" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121584 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121618 4784 cpu_manager.go:414] "containerMap: removing container" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121626 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121635 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121642 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.121652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121651 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121658 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121673 4784 cpu_manager.go:414] "containerMap: removing container" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121680 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121804 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121817 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121830 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121839 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121848 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f1f06-ca99-4bbd-a290-b466927b15a4" containerName="cleanup" Nov 04 02:45:47.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.121857 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="283d2e6a-22d3-4aa9-95d5-d5069aa2024a" containerName="cleanup" Nov 04 02:45:47.124296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.124275 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.158820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.158777 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.159108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.159077 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.159272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.159124 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.159272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.159244 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5rf\" (UniqueName: \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.159416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.159317 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.159530 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.159499 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.194514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.194477 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 02:45:47.230941 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.230896 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"default-dockercfg-2rh2k\"" Nov 04 02:45:47.230941 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.230931 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"simulator-w0-fe4e0c-kserve-self-signed-certs\"" Nov 04 02:45:47.230941 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.230941 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"kube-root-ca.crt\"" Nov 04 02:45:47.231215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.231166 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"openshift-service-ca.crt\"" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261124 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261241 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261284 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261312 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261421 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5rf\" (UniqueName: \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261502 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261661 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.261946 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.262626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.262177 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.265501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.265476 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.266962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.266936 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.324771 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.324734 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5rf\" (UniqueName: \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\") pod \"simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.435625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.435591 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:47.667784 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.667752 4784 generic.go:356] "Generic (PLEG): container finished" podID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerID="cdb281b33af22ea6baecae9164d34defff666dc1bd176aeff6a2cf7c0688268d" exitCode=0 Nov 04 02:45:47.667976 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.667826 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerDied","Data":"cdb281b33af22ea6baecae9164d34defff666dc1bd176aeff6a2cf7c0688268d"} Nov 04 02:45:47.668307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.668288 4784 scope.go:122] "RemoveContainer" containerID="cdb281b33af22ea6baecae9164d34defff666dc1bd176aeff6a2cf7c0688268d" Nov 04 02:45:47.669481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669456 4784 generic.go:356] "Generic (PLEG): container finished" podID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerID="99174578f0e7cde51294e91604875b1a1c65e2c63d5dc0b8cf3641bc618f159a" exitCode=0 Nov 04 02:45:47.669588 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669528 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerDied","Data":"99174578f0e7cde51294e91604875b1a1c65e2c63d5dc0b8cf3641bc618f159a"} Nov 04 02:45:47.669653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669630 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.669709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669692 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.669761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669724 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.669917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.669900 4784 scope.go:122] "RemoveContainer" containerID="99174578f0e7cde51294e91604875b1a1c65e2c63d5dc0b8cf3641bc618f159a" Nov 04 02:45:47.670976 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.670948 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.671068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.671027 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.671129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.671078 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:45:47.828972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:47.823163 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 02:45:47.856812 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:45:47.856780 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3727622_6296_404c_b05d_cf537d65ea4b.slice/crio-1af9875535b9dc5c52ee279a4b32219d7ee7d57fe72ba23ffda0cd5b5504ec48 WatchSource:0}: Error finding container 1af9875535b9dc5c52ee279a4b32219d7ee7d57fe72ba23ffda0cd5b5504ec48: Status 404 returned error can't find the container with id 1af9875535b9dc5c52ee279a4b32219d7ee7d57fe72ba23ffda0cd5b5504ec48 Nov 04 02:45:48.019743 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:48.019704 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:45:48.710796 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:48.710764 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerStarted","Data":"de62a7c945b370b0a8f484cead3586bb9a09b9f87bcaa78bad12a595063ebed0"} Nov 04 02:45:48.713489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:48.712822 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" event={"ID":"f3727622-6296-404c-b05d-cf537d65ea4b","Type":"ContainerStarted","Data":"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47"} Nov 04 02:45:48.713489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:48.712857 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" event={"ID":"f3727622-6296-404c-b05d-cf537d65ea4b","Type":"ContainerStarted","Data":"1af9875535b9dc5c52ee279a4b32219d7ee7d57fe72ba23ffda0cd5b5504ec48"} Nov 04 02:45:48.715287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:48.715258 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerStarted","Data":"461550caafa986b2ec9eb85ec794428a77fa99019112cac2131f2efe9c4d944c"} Nov 04 02:45:49.026012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.025923 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:45:49.124281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.124162 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podStartSLOduration=2.124147095 podStartE2EDuration="2.124147095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:48.998442204 +0000 UTC m=+795.964943210" watchObservedRunningTime="2026-11-04 02:45:49.124147095 +0000 UTC m=+796.090648090" Nov 04 02:45:49.435891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.435865 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:49.540996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.540963 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:45:49.542162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.542136 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:45:49.542235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.542213 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 02:45:49.614517 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.614479 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:45:49.615636 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.615609 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:45:49.615702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:49.615664 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 02:45:50.541419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:50.541378 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:45:50.541910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:50.541441 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 02:45:50.614724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:50.614687 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:45:50.614890 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:50.614741 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 02:45:51.541074 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:51.541042 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:45:51.541346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:51.541325 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 02:45:51.614553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:51.614523 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:45:51.615023 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:51.614995 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 02:45:52.541086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:52.541049 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:45:52.541296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:52.541113 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 02:45:52.614728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:52.614681 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:45:52.615141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:52.614762 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 02:45:53.540885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:53.540848 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:45:53.541219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:53.541162 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 02:45:53.615236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:53.615171 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:45:53.615658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:53.615279 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 02:45:54.545993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:54.545963 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:45:54.547043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:54.547022 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 02:45:54.619017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:54.618990 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:45:54.619514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:54.619370 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:45:54.620254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:54.620234 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 02:45:56.785414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.785382 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 02:45:56.789794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.789753 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.875224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.875167 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 02:45:56.895166 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.895133 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"kube-root-ca.crt\"" Nov 04 02:45:56.895166 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.895156 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"default-dockercfg-fr85m\"" Nov 04 02:45:56.895379 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.895222 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"openshift-service-ca.crt\"" Nov 04 02:45:56.895379 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.895245 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"simulator-w2-a0bcbc-kserve-self-signed-certs\"" Nov 04 02:45:56.934052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934005 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.934256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934176 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgh5x\" (UniqueName: \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.934338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934304 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.934378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934365 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.934496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934391 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:56.934496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:56.934414 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035475 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035383 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035475 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035447 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgh5x\" (UniqueName: \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035475 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035475 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035786 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035506 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035786 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035527 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035786 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035556 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035865 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.035936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035871 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.036032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.035937 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.038039 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.038016 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.038354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.038334 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.098099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.098065 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgh5x\" (UniqueName: \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\") pod \"simulator-w2-a0bcbc-kserve-68759db55d-zlpr9\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.107047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.107022 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:45:57.436637 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.436604 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:57.449361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.449331 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:57.494245 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.494219 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 02:45:57.496435 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:45:57.496408 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd83d1144_9af1_4bf3_9a77_dab59d43cc30.slice/crio-506559081b343190265850b2066dc4c380d85441804c5af3507af69ce2cef6ee WatchSource:0}: Error finding container 506559081b343190265850b2066dc4c380d85441804c5af3507af69ce2cef6ee: Status 404 returned error can't find the container with id 506559081b343190265850b2066dc4c380d85441804c5af3507af69ce2cef6ee Nov 04 02:45:57.776381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.776278 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" event={"ID":"d83d1144-9af1-4bf3-9a77-dab59d43cc30","Type":"ContainerStarted","Data":"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f"} Nov 04 02:45:57.776381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.776330 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" event={"ID":"d83d1144-9af1-4bf3-9a77-dab59d43cc30","Type":"ContainerStarted","Data":"506559081b343190265850b2066dc4c380d85441804c5af3507af69ce2cef6ee"} Nov 04 02:45:57.789743 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.789716 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:45:57.896849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:57.896795 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podStartSLOduration=1.896780731 podStartE2EDuration="1.896780731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:57.895371903 +0000 UTC m=+804.861872896" watchObservedRunningTime="2026-11-04 02:45:57.896780731 +0000 UTC m=+804.863281722" Nov 04 02:45:58.107423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:45:58.107341 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:46:06.400610 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:46:06.400560 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:46:07.108080 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:07.108027 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:46:07.121281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:07.121250 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:46:07.853280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:07.853253 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:46:20.776021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.775977 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 02:46:20.780255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.780229 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811035 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811170 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811228 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlwc\" (UniqueName: \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811247 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811509 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811344 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.811509 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.811367 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.885549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.885512 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3b7476\"/\"kube-root-ca.crt\"" Nov 04 02:46:20.885786 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.885515 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3b7476\"/\"openshift-service-ca.crt\"" Nov 04 02:46:20.885878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.885593 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b7476\"/\"default-dockercfg-q7bsj\"" Nov 04 02:46:20.886052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.885672 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b7476\"/\"simulator-w5-3b7476-kserve-self-signed-certs\"" Nov 04 02:46:20.896618 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.896587 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 02:46:20.911995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.911956 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlwc\" (UniqueName: \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.911995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.911999 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912050 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912076 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912162 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912345 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912587 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912558 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912587 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912585 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.912797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.912610 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.914423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.914399 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.914820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.914803 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:20.976792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:20.976762 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlwc\" (UniqueName: \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\") pod \"simulator-w5-3b7476-kserve-799f77bb97-tcpgt\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:21.095428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:21.095329 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:21.473931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:21.473905 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 02:46:21.474443 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:46:21.474415 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d67afa4_bd4c_4e78_bdd5_ae39c6d45e02.slice/crio-f11f95288272d392e0e28564d8462a3c792df987df861c652a7b963bd54a3247 WatchSource:0}: Error finding container f11f95288272d392e0e28564d8462a3c792df987df861c652a7b963bd54a3247: Status 404 returned error can't find the container with id f11f95288272d392e0e28564d8462a3c792df987df861c652a7b963bd54a3247 Nov 04 02:46:21.938574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:21.938537 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" event={"ID":"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02","Type":"ContainerStarted","Data":"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca"} Nov 04 02:46:21.938982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:21.938584 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" event={"ID":"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02","Type":"ContainerStarted","Data":"f11f95288272d392e0e28564d8462a3c792df987df861c652a7b963bd54a3247"} Nov 04 02:46:22.062036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:22.061984 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podStartSLOduration=2.061968908 podStartE2EDuration="2.061968908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:46:22.061172371 +0000 UTC m=+829.027673365" watchObservedRunningTime="2026-11-04 02:46:22.061968908 +0000 UTC m=+829.028469900" Nov 04 02:46:22.095725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:22.095693 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:31.095600 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:31.095543 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:31.109103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:31.109068 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:32.016548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:32.016518 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:46:36.400038 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:46:36.400002 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:46:45.444149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.444065 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 02:46:45.449215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.449170 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.494853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.494812 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.494874 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdrp\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.494907 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.494995 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.495049 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.495216 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.495286 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.495340 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.495440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.495425 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.556594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.556560 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cdcf67-openshift-default-dockercfg-pbxqp\"" Nov 04 02:46:45.559908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.559879 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 02:46:45.595999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.595962 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596011 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596136 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596217 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596297 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596335 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdrp\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596366 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596426 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596462 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596611 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596475 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596611 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596532 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596608 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.596908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.596889 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.598829 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.598796 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.599041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.599022 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.659427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.659393 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.659781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.659759 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdrp\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:45.788503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:45.788410 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:46.221342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:46.221299 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 02:46:46.221548 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:46:46.221527 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda87486b_e9f0_4206_9f05_37d685d811bd.slice/crio-5cda08020cf4c738dbf65febe2ef678421401f5b7a3a72b5bc3d39062e6e1db6 WatchSource:0}: Error finding container 5cda08020cf4c738dbf65febe2ef678421401f5b7a3a72b5bc3d39062e6e1db6: Status 404 returned error can't find the container with id 5cda08020cf4c738dbf65febe2ef678421401f5b7a3a72b5bc3d39062e6e1db6 Nov 04 02:46:46.223927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:46.223895 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:46:46.224002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:46.223963 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:46:46.224002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:46.223990 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:46:47.101943 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.101906 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" event={"ID":"da87486b-e9f0-4206-9f05-37d685d811bd","Type":"ContainerStarted","Data":"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3"} Nov 04 02:46:47.102375 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.101962 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" event={"ID":"da87486b-e9f0-4206-9f05-37d685d811bd","Type":"ContainerStarted","Data":"5cda08020cf4c738dbf65febe2ef678421401f5b7a3a72b5bc3d39062e6e1db6"} Nov 04 02:46:47.225796 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.225740 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podStartSLOduration=2.22572601 podStartE2EDuration="2.22572601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:46:47.223975233 +0000 UTC m=+854.190476240" watchObservedRunningTime="2026-11-04 02:46:47.22572601 +0000 UTC m=+854.192227000" Nov 04 02:46:47.789348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.789308 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:47.791522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.791495 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 02:46:47.791671 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:47.791545 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 02:46:48.789059 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:48.789019 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 02:46:48.789602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:48.789082 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 02:46:49.788991 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:49.788948 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 02:46:49.789173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:49.789014 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 02:46:50.789752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:50.789708 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 02:46:50.790442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:50.790410 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 02:46:51.789283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:51.789243 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 02:46:51.789555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:51.789311 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 02:46:53.769144 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:53.769103 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:53.770885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:53.770852 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:46:53.771713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:46:53.771693 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 02:47:06.492843 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:47:06.492804 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:47:09.478324 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.478273 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 02:47:09.483462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.483439 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514562 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514599 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514689 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514749 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514807 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdz45\" (UniqueName: \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.514885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.514850 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.540851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.540818 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 02:47:09.588273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.588241 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"premium-w0-fe4e0c-kserve-self-signed-certs\"" Nov 04 02:47:09.616054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616016 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616099 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdz45\" (UniqueName: \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616139 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616263 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616285 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616374 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616504 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616581 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.616704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.616660 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.618711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.618664 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.619119 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.619099 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.679140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.679100 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdz45\" (UniqueName: \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\") pod \"premium-w0-fe4e0c-kserve-dfd957b87-nsd5x\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:09.797026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:09.796935 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:10.182973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:10.182940 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 02:47:10.183624 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:47:10.183599 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5db08a49_57b2_4e07_a805_be215c07b917.slice/crio-d023b4dbe670df9d7153df46ff92fe5d765179bb504918398f1930ce79ae5179 WatchSource:0}: Error finding container d023b4dbe670df9d7153df46ff92fe5d765179bb504918398f1930ce79ae5179: Status 404 returned error can't find the container with id d023b4dbe670df9d7153df46ff92fe5d765179bb504918398f1930ce79ae5179 Nov 04 02:47:10.289624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:10.289586 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" event={"ID":"5db08a49-57b2-4e07-a805-be215c07b917","Type":"ContainerStarted","Data":"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c"} Nov 04 02:47:10.289794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:10.289637 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" event={"ID":"5db08a49-57b2-4e07-a805-be215c07b917","Type":"ContainerStarted","Data":"d023b4dbe670df9d7153df46ff92fe5d765179bb504918398f1930ce79ae5179"} Nov 04 02:47:10.411382 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:10.411328 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podStartSLOduration=1.411311891 podStartE2EDuration="1.411311891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:47:10.408012808 +0000 UTC m=+877.374513800" watchObservedRunningTime="2026-11-04 02:47:10.411311891 +0000 UTC m=+877.377812880" Nov 04 02:47:10.798039 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:10.798004 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:19.797672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:19.797636 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:19.810524 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:19.810494 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:20.363513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:20.363485 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:47:33.396094 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:33.396062 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:47:33.396585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:33.396321 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:47:33.402663 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:33.402642 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:47:33.402811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:33.402748 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:47:36.400143 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:47:36.400106 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:47:55.129070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.129030 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8"] Nov 04 02:47:55.133091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.133075 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.191054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.191020 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8"] Nov 04 02:47:55.237500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.237465 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:47:55.237500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.237465 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:47:55.237716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.237467 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:47:55.237803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.237787 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:47:55.275521 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.275487 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fzs\" (UniqueName: \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.275697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.275629 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.275697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.275681 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.376715 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.376683 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.376922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.376781 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fzs\" (UniqueName: \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.376922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.376847 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.377498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.377433 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.379333 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.379275 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.439677 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.439638 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fzs\" (UniqueName: \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\") pod \"maas-api-revoke-sub-48ffac3b-bkwk8\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.447058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.447032 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:55.824598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:55.824572 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8"] Nov 04 02:47:55.825666 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:47:55.825633 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db696c_fb00_4bcc_83f9_e3202989fd01.slice/crio-6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22 WatchSource:0}: Error finding container 6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22: Status 404 returned error can't find the container with id 6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22 Nov 04 02:47:56.591520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:56.591421 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" event={"ID":"d5db696c-fb00-4bcc-83f9-e3202989fd01","Type":"ContainerStarted","Data":"54f613292b07fdfe4267522e2e2f724fc701be59bf22da10a2bddfdf0060d8ad"} Nov 04 02:47:56.591520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:56.591482 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" event={"ID":"d5db696c-fb00-4bcc-83f9-e3202989fd01","Type":"ContainerStarted","Data":"6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22"} Nov 04 02:47:56.710530 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:56.710473 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" podStartSLOduration=1.71045724 podStartE2EDuration="1.71045724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:47:56.70878695 +0000 UTC m=+923.675287949" watchObservedRunningTime="2026-11-04 02:47:56.71045724 +0000 UTC m=+923.676958265" Nov 04 02:47:57.600032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:57.599996 4784 generic.go:356] "Generic (PLEG): container finished" podID="d5db696c-fb00-4bcc-83f9-e3202989fd01" containerID="54f613292b07fdfe4267522e2e2f724fc701be59bf22da10a2bddfdf0060d8ad" exitCode=0 Nov 04 02:47:57.600458 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:57.600046 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" event={"ID":"d5db696c-fb00-4bcc-83f9-e3202989fd01","Type":"ContainerDied","Data":"54f613292b07fdfe4267522e2e2f724fc701be59bf22da10a2bddfdf0060d8ad"} Nov 04 02:47:58.769688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.769659 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:58.917735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.917696 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\" (UniqueName: \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\") pod \"d5db696c-fb00-4bcc-83f9-e3202989fd01\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " Nov 04 02:47:58.917925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.917757 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\") pod \"d5db696c-fb00-4bcc-83f9-e3202989fd01\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " Nov 04 02:47:58.918005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.917923 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\") pod \"d5db696c-fb00-4bcc-83f9-e3202989fd01\" (UID: \"d5db696c-fb00-4bcc-83f9-e3202989fd01\") " Nov 04 02:47:58.919054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.919019 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca" pod "d5db696c-fb00-4bcc-83f9-e3202989fd01" (UID: "d5db696c-fb00-4bcc-83f9-e3202989fd01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:47:58.921664 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.921591 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca" pod "d5db696c-fb00-4bcc-83f9-e3202989fd01" (UID: "d5db696c-fb00-4bcc-83f9-e3202989fd01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:47:58.921933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:58.921907 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs" pod "d5db696c-fb00-4bcc-83f9-e3202989fd01" (UID: "d5db696c-fb00-4bcc-83f9-e3202989fd01"). InnerVolumeSpecName "kube-api-access-n5fzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:47:59.019450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.019414 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:47:59.019450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.019443 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fzs\" (UniqueName: \"kubernetes.io/projected/d5db696c-fb00-4bcc-83f9-e3202989fd01-kube-api-access-n5fzs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:47:59.019450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.019454 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5db696c-fb00-4bcc-83f9-e3202989fd01-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:47:59.616704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.616606 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" Nov 04 02:47:59.616704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.616624 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8" event={"ID":"d5db696c-fb00-4bcc-83f9-e3202989fd01","Type":"ContainerDied","Data":"6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22"} Nov 04 02:47:59.616704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:47:59.616676 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6404bc9b8631e5aa9f4eb39ed9a79410f77e78c1eeca216a8cbeb04a876b4e22" Nov 04 02:48:01.887900 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.887866 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 02:48:01.888479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.888464 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d5db696c-fb00-4bcc-83f9-e3202989fd01" containerName="revoke-keys" Nov 04 02:48:01.888535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.888481 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5db696c-fb00-4bcc-83f9-e3202989fd01" containerName="revoke-keys" Nov 04 02:48:01.888594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.888585 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5db696c-fb00-4bcc-83f9-e3202989fd01" containerName="revoke-keys" Nov 04 02:48:01.892051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.892036 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:01.997753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.997718 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"kube-root-ca.crt\"" Nov 04 02:48:01.997753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.997727 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"openshift-service-ca.crt\"" Nov 04 02:48:01.997978 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.997811 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"default-dockercfg-9d9b9\"" Nov 04 02:48:01.997978 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:01.997900 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"premium-w1-e516ff-kserve-self-signed-certs\"" Nov 04 02:48:02.001247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.001224 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 02:48:02.056956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.056923 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.056956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.056972 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.057251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.057085 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.057251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.057118 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.057251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.057136 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.057361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.057317 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjmt\" (UniqueName: \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.158688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158598 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjmt\" (UniqueName: \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.158856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158712 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.158856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158735 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.158856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158765 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.158856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158842 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.159036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.158882 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.159175 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.159155 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.159292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.159178 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.159348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.159292 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.161427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.161403 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.161682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.161661 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.221485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.221456 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjmt\" (UniqueName: \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\") pod \"premium-w1-e516ff-kserve-5684f77c78-vqlz5\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.505047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.505014 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:02.887083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:02.887038 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 02:48:03.643133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:03.643095 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" event={"ID":"2fa6c3fe-7c55-4688-b225-423d82cae782","Type":"ContainerStarted","Data":"ad8d10bf39ddd85083df91455f829202d3bd81bbcd7f4d21e290df6de64b2885"} Nov 04 02:48:03.643588 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:03.643143 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" event={"ID":"2fa6c3fe-7c55-4688-b225-423d82cae782","Type":"ContainerStarted","Data":"b7070a54d5fd7dd2b1c99f208199a9cbbe777426969913100b0a1896453e646e"} Nov 04 02:48:03.763891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:03.763828 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podStartSLOduration=2.763814211 podStartE2EDuration="2.763814211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:03.761404978 +0000 UTC m=+930.727905969" watchObservedRunningTime="2026-11-04 02:48:03.763814211 +0000 UTC m=+930.730315203" Nov 04 02:48:04.505832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:04.505800 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:06.400135 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:48:06.400099 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:48:10.649786 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.649752 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr"] Nov 04 02:48:10.654214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.654171 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.711141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.711112 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr"] Nov 04 02:48:10.758322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.758289 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:48:10.758322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.758294 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:48:10.758322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.758317 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:48:10.758573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.758346 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:48:10.761972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.761950 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.762029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.762011 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.762181 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.762164 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jpm\" (UniqueName: \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.862953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.862919 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jpm\" (UniqueName: \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.863152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.862998 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.863152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.863065 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.863756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.863733 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.865577 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.865554 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.925283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.925253 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jpm\" (UniqueName: \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\") pod \"maas-api-revoke-sub-101d9216-n6gsr\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:10.968516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:10.968470 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:11.350363 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:48:11.350331 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207741ba_6201_43b0_a615_873051f60d97.slice/crio-2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020 WatchSource:0}: Error finding container 2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020: Status 404 returned error can't find the container with id 2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020 Nov 04 02:48:11.350494 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:11.350472 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr"] Nov 04 02:48:11.703116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:11.703077 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" event={"ID":"207741ba-6201-43b0-a615-873051f60d97","Type":"ContainerStarted","Data":"b8c23407319a051f63aaf2efdad0ce735d3731b54fbfde5d59334bfe04a22708"} Nov 04 02:48:11.703623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:11.703165 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" event={"ID":"207741ba-6201-43b0-a615-873051f60d97","Type":"ContainerStarted","Data":"2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020"} Nov 04 02:48:11.823014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:11.822958 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" podStartSLOduration=1.822943908 podStartE2EDuration="1.822943908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:11.819874024 +0000 UTC m=+938.786375020" watchObservedRunningTime="2026-11-04 02:48:11.822943908 +0000 UTC m=+938.789444900" Nov 04 02:48:12.505889 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:12.505801 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:12.518931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:12.518906 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:12.711503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:12.711458 4784 generic.go:356] "Generic (PLEG): container finished" podID="207741ba-6201-43b0-a615-873051f60d97" containerID="b8c23407319a051f63aaf2efdad0ce735d3731b54fbfde5d59334bfe04a22708" exitCode=0 Nov 04 02:48:12.711905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:12.711550 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" event={"ID":"207741ba-6201-43b0-a615-873051f60d97","Type":"ContainerDied","Data":"b8c23407319a051f63aaf2efdad0ce735d3731b54fbfde5d59334bfe04a22708"} Nov 04 02:48:12.724521 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:12.724490 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:48:13.879401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.879375 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:13.897635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.897603 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\" (UniqueName: \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\") pod \"207741ba-6201-43b0-a615-873051f60d97\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " Nov 04 02:48:13.897795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.897679 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\") pod \"207741ba-6201-43b0-a615-873051f60d97\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " Nov 04 02:48:13.897795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.897732 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\") pod \"207741ba-6201-43b0-a615-873051f60d97\" (UID: \"207741ba-6201-43b0-a615-873051f60d97\") " Nov 04 02:48:13.898066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.898042 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca" pod "207741ba-6201-43b0-a615-873051f60d97" (UID: "207741ba-6201-43b0-a615-873051f60d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:48:13.898261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.898240 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207741ba-6201-43b0-a615-873051f60d97-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:13.899917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.899890 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca" pod "207741ba-6201-43b0-a615-873051f60d97" (UID: "207741ba-6201-43b0-a615-873051f60d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:48:13.899996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:13.899975 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm" pod "207741ba-6201-43b0-a615-873051f60d97" (UID: "207741ba-6201-43b0-a615-873051f60d97"). InnerVolumeSpecName "kube-api-access-d6jpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:48:14.000082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:14.000000 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6jpm\" (UniqueName: \"kubernetes.io/projected/207741ba-6201-43b0-a615-873051f60d97-kube-api-access-d6jpm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:14.000082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:14.000040 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207741ba-6201-43b0-a615-873051f60d97-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:14.728591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:14.728564 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" Nov 04 02:48:14.728763 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:14.728564 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr" event={"ID":"207741ba-6201-43b0-a615-873051f60d97","Type":"ContainerDied","Data":"2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020"} Nov 04 02:48:14.728763 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:14.728689 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fcd6cddb40f00cb0e43acc139f75124cd5d4e6cae3efde051834d15a69ca020" Nov 04 02:48:26.391734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.391698 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6"] Nov 04 02:48:26.392264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.392255 4784 cpu_manager.go:414] "containerMap: removing container" podUID="207741ba-6201-43b0-a615-873051f60d97" containerName="revoke-keys" Nov 04 02:48:26.392312 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.392267 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207741ba-6201-43b0-a615-873051f60d97" containerName="revoke-keys" Nov 04 02:48:26.392382 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.392371 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="207741ba-6201-43b0-a615-873051f60d97" containerName="revoke-keys" Nov 04 02:48:26.397060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.397035 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.454520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.454480 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6"] Nov 04 02:48:26.468115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.468084 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlw27\" (UniqueName: \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.468339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.468150 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.468428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.468373 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.502126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.502095 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:48:26.502126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.502121 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:48:26.553156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.553130 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:48:26.553332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.553247 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:48:26.569745 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.569714 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.569889 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.569843 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.569969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.569952 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlw27\" (UniqueName: \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.570513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.570464 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.572235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.572183 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.632592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.632563 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlw27\" (UniqueName: \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\") pod \"maas-api-revoke-sub-e95d06e4-fnrp6\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:26.714383 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:26.714351 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:27.095845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:27.095809 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6"] Nov 04 02:48:27.096592 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:48:27.096567 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68de16b5_94b2_4587_8b70_60f42fdab656.slice/crio-16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542 WatchSource:0}: Error finding container 16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542: Status 404 returned error can't find the container with id 16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542 Nov 04 02:48:27.821417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:27.821381 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" event={"ID":"68de16b5-94b2-4587-8b70-60f42fdab656","Type":"ContainerStarted","Data":"c0302cbc773840e852b0b8a7ccad6c80fa1823d028dfc1d5f0a74ff28b4aebd1"} Nov 04 02:48:27.821850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:27.821429 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" event={"ID":"68de16b5-94b2-4587-8b70-60f42fdab656","Type":"ContainerStarted","Data":"16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542"} Nov 04 02:48:27.942993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:27.942905 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" podStartSLOduration=1.94288979 podStartE2EDuration="1.94288979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:27.938992365 +0000 UTC m=+954.905493358" watchObservedRunningTime="2026-11-04 02:48:27.94288979 +0000 UTC m=+954.909390782" Nov 04 02:48:28.830409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:28.830370 4784 generic.go:356] "Generic (PLEG): container finished" podID="68de16b5-94b2-4587-8b70-60f42fdab656" containerID="c0302cbc773840e852b0b8a7ccad6c80fa1823d028dfc1d5f0a74ff28b4aebd1" exitCode=0 Nov 04 02:48:28.830832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:28.830463 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" event={"ID":"68de16b5-94b2-4587-8b70-60f42fdab656","Type":"ContainerDied","Data":"c0302cbc773840e852b0b8a7ccad6c80fa1823d028dfc1d5f0a74ff28b4aebd1"} Nov 04 02:48:29.986731 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:29.986704 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:30.111069 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.110987 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\") pod \"68de16b5-94b2-4587-8b70-60f42fdab656\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " Nov 04 02:48:30.111069 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.111038 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\" (UniqueName: \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\") pod \"68de16b5-94b2-4587-8b70-60f42fdab656\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " Nov 04 02:48:30.111321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.111107 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\") pod \"68de16b5-94b2-4587-8b70-60f42fdab656\" (UID: \"68de16b5-94b2-4587-8b70-60f42fdab656\") " Nov 04 02:48:30.111389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.111363 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca" pod "68de16b5-94b2-4587-8b70-60f42fdab656" (UID: "68de16b5-94b2-4587-8b70-60f42fdab656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:48:30.111638 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.111617 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:30.113635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.113606 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27" pod "68de16b5-94b2-4587-8b70-60f42fdab656" (UID: "68de16b5-94b2-4587-8b70-60f42fdab656"). InnerVolumeSpecName "kube-api-access-nlw27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:48:30.117310 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.117279 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca" pod "68de16b5-94b2-4587-8b70-60f42fdab656" (UID: "68de16b5-94b2-4587-8b70-60f42fdab656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:48:30.212512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.212479 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlw27\" (UniqueName: \"kubernetes.io/projected/68de16b5-94b2-4587-8b70-60f42fdab656-kube-api-access-nlw27\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:30.212512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.212509 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de16b5-94b2-4587-8b70-60f42fdab656-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:30.845147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.845113 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" event={"ID":"68de16b5-94b2-4587-8b70-60f42fdab656","Type":"ContainerDied","Data":"16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542"} Nov 04 02:48:30.845147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.845134 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6" Nov 04 02:48:30.845398 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:30.845211 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16fea803b154044e6facce6b4d3b1385b27d0c33b4ccaa5567ecb6be27ea9542" Nov 04 02:48:36.402950 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:48:36.402914 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:48:46.725820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.725787 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h"] Nov 04 02:48:46.726420 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.726405 4784 cpu_manager.go:414] "containerMap: removing container" podUID="68de16b5-94b2-4587-8b70-60f42fdab656" containerName="revoke-keys" Nov 04 02:48:46.726474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.726422 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68de16b5-94b2-4587-8b70-60f42fdab656" containerName="revoke-keys" Nov 04 02:48:46.726515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.726512 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="68de16b5-94b2-4587-8b70-60f42fdab656" containerName="revoke-keys" Nov 04 02:48:46.729838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.729820 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.784110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.784070 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:48:46.790844 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.790816 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2d4w\" (UniqueName: \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.791007 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.790851 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.791086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.791064 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.835054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.835020 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:48:46.835054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.835044 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:48:46.835054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.835033 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:48:46.838640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.838614 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h"] Nov 04 02:48:46.896211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.893934 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.896211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.894058 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2d4w\" (UniqueName: \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.896211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.894095 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.896211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.895030 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.902965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.902929 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:46.956665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:46.956628 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2d4w\" (UniqueName: \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\") pod \"maas-api-revoke-sub-de56d712-2ll7h\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:47.008182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.008092 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 02:48:47.012461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.012440 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.043352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.043322 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:47.068368 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.068337 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e516ff-openshift-default-dockercfg-5x2gt\"" Nov 04 02:48:47.073040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.073012 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 02:48:47.096716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.096684 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.096716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.096722 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.096981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.096908 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.096981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.096950 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.097098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.097049 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.097156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.097144 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.097277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.097258 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.097363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.097326 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7d2p\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.097448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.097431 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200346 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200464 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200531 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7d2p\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200616 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200698 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200726 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200792 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200816 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200819 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.200898 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.201387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.201170 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.202141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.201449 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.202141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.201896 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.202141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.202055 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.206212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.204645 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.214239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.207734 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.266078 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.265970 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7d2p\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.269855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.269768 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.328022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.327989 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:47.453517 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.453431 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h"] Nov 04 02:48:47.458656 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:48:47.458629 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9779eee6_26df_49b3_9350_180c17ef2675.slice/crio-d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0 WatchSource:0}: Error finding container d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0: Status 404 returned error can't find the container with id d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0 Nov 04 02:48:47.715602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.715580 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 02:48:47.758425 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:48:47.758382 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08292c14_0f3c_4af5_8972_94578b37990b.slice/crio-08de33a3d40da8cda142acb53b084a1dfa90c55fa2d636edbcba43647b131853 WatchSource:0}: Error finding container 08de33a3d40da8cda142acb53b084a1dfa90c55fa2d636edbcba43647b131853: Status 404 returned error can't find the container with id 08de33a3d40da8cda142acb53b084a1dfa90c55fa2d636edbcba43647b131853 Nov 04 02:48:47.761139 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.761106 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:48:47.761275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.761178 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:48:47.761275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.761228 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:48:47.980965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.980861 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" event={"ID":"9779eee6-26df-49b3-9350-180c17ef2675","Type":"ContainerStarted","Data":"cbc3615fa1394be5eb0e17209bebe6cbb98f82044d18f4f49d0c16f69f66737a"} Nov 04 02:48:47.980965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.980925 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" event={"ID":"9779eee6-26df-49b3-9350-180c17ef2675","Type":"ContainerStarted","Data":"d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0"} Nov 04 02:48:47.982850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.982820 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" event={"ID":"08292c14-0f3c-4af5-8972-94578b37990b","Type":"ContainerStarted","Data":"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18"} Nov 04 02:48:47.982962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:47.982875 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" event={"ID":"08292c14-0f3c-4af5-8972-94578b37990b","Type":"ContainerStarted","Data":"08de33a3d40da8cda142acb53b084a1dfa90c55fa2d636edbcba43647b131853"} Nov 04 02:48:48.099559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.099503 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" podStartSLOduration=2.099489373 podStartE2EDuration="2.099489373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:48.098157613 +0000 UTC m=+975.064658606" watchObservedRunningTime="2026-11-04 02:48:48.099489373 +0000 UTC m=+975.065990364" Nov 04 02:48:48.224579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.224514 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podStartSLOduration=2.224495215 podStartE2EDuration="2.224495215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:48.223637593 +0000 UTC m=+975.190138585" watchObservedRunningTime="2026-11-04 02:48:48.224495215 +0000 UTC m=+975.190996208" Nov 04 02:48:48.329146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.329064 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:48.330650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.330624 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:48.330794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.330681 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:48.991527 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.991494 4784 generic.go:356] "Generic (PLEG): container finished" podID="9779eee6-26df-49b3-9350-180c17ef2675" containerID="cbc3615fa1394be5eb0e17209bebe6cbb98f82044d18f4f49d0c16f69f66737a" exitCode=0 Nov 04 02:48:48.991932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:48.991572 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" event={"ID":"9779eee6-26df-49b3-9350-180c17ef2675","Type":"ContainerDied","Data":"cbc3615fa1394be5eb0e17209bebe6cbb98f82044d18f4f49d0c16f69f66737a"} Nov 04 02:48:49.328466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:49.328376 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:49.328466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:49.328434 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:50.237406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.237379 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:50.328874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.328788 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:50.328874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.328845 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:50.349552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.349517 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\" (UniqueName: \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\") pod \"9779eee6-26df-49b3-9350-180c17ef2675\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " Nov 04 02:48:50.349753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.349737 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\") pod \"9779eee6-26df-49b3-9350-180c17ef2675\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " Nov 04 02:48:50.349838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.349790 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\") pod \"9779eee6-26df-49b3-9350-180c17ef2675\" (UID: \"9779eee6-26df-49b3-9350-180c17ef2675\") " Nov 04 02:48:50.350282 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.350168 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca" pod "9779eee6-26df-49b3-9350-180c17ef2675" (UID: "9779eee6-26df-49b3-9350-180c17ef2675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:48:50.351846 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.351823 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w" pod "9779eee6-26df-49b3-9350-180c17ef2675" (UID: "9779eee6-26df-49b3-9350-180c17ef2675"). InnerVolumeSpecName "kube-api-access-f2d4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:48:50.351961 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.351847 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca" pod "9779eee6-26df-49b3-9350-180c17ef2675" (UID: "9779eee6-26df-49b3-9350-180c17ef2675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:48:50.451302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.451268 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:50.451302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.451300 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9779eee6-26df-49b3-9350-180c17ef2675-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:50.451302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:50.451311 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2d4w\" (UniqueName: \"kubernetes.io/projected/9779eee6-26df-49b3-9350-180c17ef2675-kube-api-access-f2d4w\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:48:51.007531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:51.007490 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" Nov 04 02:48:51.007531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:51.007523 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h" event={"ID":"9779eee6-26df-49b3-9350-180c17ef2675","Type":"ContainerDied","Data":"d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0"} Nov 04 02:48:51.007761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:51.007575 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4f694627a0becb36813ba96284ee82ae4a9580b8a51ddf9c78e01ff4e45a3a0" Nov 04 02:48:51.328815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:51.328732 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:51.328815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:51.328787 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:52.328759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:52.328721 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:52.328928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:52.328811 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:53.329177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:53.329141 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:53.329666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:53.329234 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:54.328819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:54.328785 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 02:48:54.328991 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:54.328858 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 02:48:55.337685 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:55.337655 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:55.338126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:55.337953 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:55.338706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:55.338686 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 02:48:57.551092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.551056 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr"] Nov 04 02:48:57.551684 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.551669 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9779eee6-26df-49b3-9350-180c17ef2675" containerName="revoke-keys" Nov 04 02:48:57.551729 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.551685 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9779eee6-26df-49b3-9350-180c17ef2675" containerName="revoke-keys" Nov 04 02:48:57.551827 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.551818 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9779eee6-26df-49b3-9350-180c17ef2675" containerName="revoke-keys" Nov 04 02:48:57.556000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.555981 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.612269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.612234 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr"] Nov 04 02:48:57.654448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.654410 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9l4\" (UniqueName: \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.654642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.654462 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.654642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.654492 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.659272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.659224 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:48:57.659272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.659248 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:48:57.659434 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.659248 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:48:57.660332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.660315 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:48:57.755717 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.755682 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9l4\" (UniqueName: \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.755949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.755733 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.755949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.755760 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.756492 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.756464 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.758429 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.758407 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.818161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.818088 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9l4\" (UniqueName: \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\") pod \"maas-api-revoke-sub-e36fb0a0-h5hsr\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:57.870168 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:57.870132 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:48:58.250290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:58.250266 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr"] Nov 04 02:48:58.251357 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:48:58.251321 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f50898_36e2_43a4_bcd7_2c0053587d60.slice/crio-5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667 WatchSource:0}: Error finding container 5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667: Status 404 returned error can't find the container with id 5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667 Nov 04 02:48:59.065289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:59.065245 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" event={"ID":"d7f50898-36e2-43a4-bcd7-2c0053587d60","Type":"ContainerStarted","Data":"2b0818927bb8e8c1d9e7cb01611492a47e426b0e01feecea10986e940adef9c6"} Nov 04 02:48:59.065787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:59.065308 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" event={"ID":"d7f50898-36e2-43a4-bcd7-2c0053587d60","Type":"ContainerStarted","Data":"5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667"} Nov 04 02:48:59.186654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:48:59.186592 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" podStartSLOduration=2.186574112 podStartE2EDuration="2.186574112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:59.18295849 +0000 UTC m=+986.149459480" watchObservedRunningTime="2026-11-04 02:48:59.186574112 +0000 UTC m=+986.153075105" Nov 04 02:49:00.074969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:00.074933 4784 generic.go:356] "Generic (PLEG): container finished" podID="d7f50898-36e2-43a4-bcd7-2c0053587d60" containerID="2b0818927bb8e8c1d9e7cb01611492a47e426b0e01feecea10986e940adef9c6" exitCode=0 Nov 04 02:49:00.075394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:00.074973 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" event={"ID":"d7f50898-36e2-43a4-bcd7-2c0053587d60","Type":"ContainerDied","Data":"2b0818927bb8e8c1d9e7cb01611492a47e426b0e01feecea10986e940adef9c6"} Nov 04 02:49:01.418361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.418331 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:49:01.534876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.534842 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\") pod \"d7f50898-36e2-43a4-bcd7-2c0053587d60\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " Nov 04 02:49:01.535088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.534971 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\" (UniqueName: \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\") pod \"d7f50898-36e2-43a4-bcd7-2c0053587d60\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " Nov 04 02:49:01.535088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.535065 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\") pod \"d7f50898-36e2-43a4-bcd7-2c0053587d60\" (UID: \"d7f50898-36e2-43a4-bcd7-2c0053587d60\") " Nov 04 02:49:01.535522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.535490 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca" pod "d7f50898-36e2-43a4-bcd7-2c0053587d60" (UID: "d7f50898-36e2-43a4-bcd7-2c0053587d60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:01.537146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.537117 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca" pod "d7f50898-36e2-43a4-bcd7-2c0053587d60" (UID: "d7f50898-36e2-43a4-bcd7-2c0053587d60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:01.537146 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.537126 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4" pod "d7f50898-36e2-43a4-bcd7-2c0053587d60" (UID: "d7f50898-36e2-43a4-bcd7-2c0053587d60"). InnerVolumeSpecName "kube-api-access-vf9l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:01.636803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.636704 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:01.636803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.636752 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9l4\" (UniqueName: \"kubernetes.io/projected/d7f50898-36e2-43a4-bcd7-2c0053587d60-kube-api-access-vf9l4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:01.636803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:01.636768 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f50898-36e2-43a4-bcd7-2c0053587d60-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:02.097901 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:02.097869 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" Nov 04 02:49:02.098091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:02.097905 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr" event={"ID":"d7f50898-36e2-43a4-bcd7-2c0053587d60","Type":"ContainerDied","Data":"5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667"} Nov 04 02:49:02.098091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:02.098004 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec2800a530240165ce0d4a4251a9e5a28247ebd4146cd20c0194f68eff97667" Nov 04 02:49:06.400206 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:49:06.400152 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:49:06.593589 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.593556 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6"] Nov 04 02:49:06.594142 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.594127 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d7f50898-36e2-43a4-bcd7-2c0053587d60" containerName="revoke-keys" Nov 04 02:49:06.594234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.594144 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f50898-36e2-43a4-bcd7-2c0053587d60" containerName="revoke-keys" Nov 04 02:49:06.594311 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.594300 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f50898-36e2-43a4-bcd7-2c0053587d60" containerName="revoke-keys" Nov 04 02:49:06.598094 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.598063 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.655131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.655047 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6"] Nov 04 02:49:06.701931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.701894 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:49:06.701931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.701920 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:49:06.702141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.701925 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:49:06.702141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.702090 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:49:06.702587 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.702568 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.702634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.702611 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4r9\" (UniqueName: \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.702802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.702781 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.803803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.803768 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.804029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.803872 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.804029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.803924 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4r9\" (UniqueName: \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.804611 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.804578 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.806353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.806335 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.866327 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.866291 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4r9\" (UniqueName: \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\") pod \"maas-api-revoke-sub-119bc78b-sqtl6\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:06.912545 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:06.912467 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:07.291912 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:07.291877 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa15464_02c6_4231_b818_3ff6c1ba5c4c.slice/crio-9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0 WatchSource:0}: Error finding container 9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0: Status 404 returned error can't find the container with id 9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0 Nov 04 02:49:07.292068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:07.292021 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6"] Nov 04 02:49:08.141885 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:08.141858 4784 generic.go:356] "Generic (PLEG): container finished" podID="9fa15464-02c6-4231-b818-3ff6c1ba5c4c" containerID="8d5a3ad0fa950e6dbc791ddc96b1ae4fe0db1f29eaae31d86b332d8116104033" exitCode=0 Nov 04 02:49:08.142264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:08.141934 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" event={"ID":"9fa15464-02c6-4231-b818-3ff6c1ba5c4c","Type":"ContainerDied","Data":"8d5a3ad0fa950e6dbc791ddc96b1ae4fe0db1f29eaae31d86b332d8116104033"} Nov 04 02:49:08.142264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:08.141989 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" event={"ID":"9fa15464-02c6-4231-b818-3ff6c1ba5c4c","Type":"ContainerStarted","Data":"9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0"} Nov 04 02:49:09.317576 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.317553 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:09.332917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.332884 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\" (UniqueName: \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\") pod \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " Nov 04 02:49:09.333075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.332948 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\") pod \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " Nov 04 02:49:09.333153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.333136 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\") pod \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\" (UID: \"9fa15464-02c6-4231-b818-3ff6c1ba5c4c\") " Nov 04 02:49:09.333519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.333492 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca" pod "9fa15464-02c6-4231-b818-3ff6c1ba5c4c" (UID: "9fa15464-02c6-4231-b818-3ff6c1ba5c4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:09.333797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.333633 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:09.335228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.335183 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca" pod "9fa15464-02c6-4231-b818-3ff6c1ba5c4c" (UID: "9fa15464-02c6-4231-b818-3ff6c1ba5c4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:09.335329 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.335236 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9" pod "9fa15464-02c6-4231-b818-3ff6c1ba5c4c" (UID: "9fa15464-02c6-4231-b818-3ff6c1ba5c4c"). InnerVolumeSpecName "kube-api-access-ql4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:09.436886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.436848 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:09.437086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:09.436897 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4r9\" (UniqueName: \"kubernetes.io/projected/9fa15464-02c6-4231-b818-3ff6c1ba5c4c-kube-api-access-ql4r9\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:10.157370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:10.157340 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" event={"ID":"9fa15464-02c6-4231-b818-3ff6c1ba5c4c","Type":"ContainerDied","Data":"9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0"} Nov 04 02:49:10.157370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:10.157356 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6" Nov 04 02:49:10.157594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:10.157387 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cdacc22a64a92fac42eb517331644756aad440622c6f8091ece7df576f4f5d0" Nov 04 02:49:23.936621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.936578 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2"] Nov 04 02:49:23.937376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.937355 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9fa15464-02c6-4231-b818-3ff6c1ba5c4c" containerName="revoke-keys" Nov 04 02:49:23.937376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.937376 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa15464-02c6-4231-b818-3ff6c1ba5c4c" containerName="revoke-keys" Nov 04 02:49:23.937562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.937549 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa15464-02c6-4231-b818-3ff6c1ba5c4c" containerName="revoke-keys" Nov 04 02:49:23.941125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.941100 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:23.999764 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:23.999733 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:49:24.002798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.002764 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2"] Nov 04 02:49:24.008840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.008759 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.009277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.009248 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wch9x\" (UniqueName: \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.009686 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.009663 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.049301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.049266 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:49:24.049483 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.049415 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:49:24.050623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.050603 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:49:24.111714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.111675 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.111898 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.111864 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wch9x\" (UniqueName: \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.111944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.111906 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.112584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.112567 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.114406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.114384 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.186666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.186595 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wch9x\" (UniqueName: \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\") pod \"maas-api-revoke-sub-13b13839-d6kz2\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.255302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.255257 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:24.642462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:24.642417 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2"] Nov 04 02:49:24.642664 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:24.642642 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e579e29_8f23_432a_9ea2_6232f656c1f6.slice/crio-f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841 WatchSource:0}: Error finding container f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841: Status 404 returned error can't find the container with id f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841 Nov 04 02:49:25.224768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.224721 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm"] Nov 04 02:49:25.228899 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.228878 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.281140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.281106 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" event={"ID":"2e579e29-8f23-432a-9ea2-6232f656c1f6","Type":"ContainerStarted","Data":"8752bca1cd6f8e518812950adfeb7b55b58a0c84a86e6e7009de56757dae21b3"} Nov 04 02:49:25.281361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.281163 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" event={"ID":"2e579e29-8f23-432a-9ea2-6232f656c1f6","Type":"ContainerStarted","Data":"f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841"} Nov 04 02:49:25.288099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.288069 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm"] Nov 04 02:49:25.327601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.327511 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.327758 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.327661 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.327876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.327853 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxvg\" (UniqueName: \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.429868 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.429835 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxvg\" (UniqueName: \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.430055 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.430025 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.430130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.430122 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.430668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.430645 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.432621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.432599 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.466544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.466485 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" podStartSLOduration=2.466469938 podStartE2EDuration="2.466469938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:25.463431159 +0000 UTC m=+1012.429932153" watchObservedRunningTime="2026-11-04 02:49:25.466469938 +0000 UTC m=+1012.432970929" Nov 04 02:49:25.492121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.492083 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxvg\" (UniqueName: \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\") pod \"maas-api-revoke-sub-3dc59bfb-cglsm\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.543000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.542965 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:25.926650 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:25.926619 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f793cc_05d3_428b_a4a2_d36eb9950f42.slice/crio-2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741 WatchSource:0}: Error finding container 2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741: Status 404 returned error can't find the container with id 2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741 Nov 04 02:49:25.927008 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:25.926981 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm"] Nov 04 02:49:26.289295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.289254 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" event={"ID":"87f793cc-05d3-428b-a4a2-d36eb9950f42","Type":"ContainerStarted","Data":"cb4c1add07fd0945645ab540e4595513da61ecb45e28735bf8edadb6d93e2cd9"} Nov 04 02:49:26.289763 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.289324 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" event={"ID":"87f793cc-05d3-428b-a4a2-d36eb9950f42","Type":"ContainerStarted","Data":"2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741"} Nov 04 02:49:26.291008 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.290980 4784 generic.go:356] "Generic (PLEG): container finished" podID="2e579e29-8f23-432a-9ea2-6232f656c1f6" containerID="8752bca1cd6f8e518812950adfeb7b55b58a0c84a86e6e7009de56757dae21b3" exitCode=0 Nov 04 02:49:26.291152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.291061 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" event={"ID":"2e579e29-8f23-432a-9ea2-6232f656c1f6","Type":"ContainerDied","Data":"8752bca1cd6f8e518812950adfeb7b55b58a0c84a86e6e7009de56757dae21b3"} Nov 04 02:49:26.410070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.410010 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" podStartSLOduration=1.409994987 podStartE2EDuration="1.409994987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:26.407118785 +0000 UTC m=+1013.373619779" watchObservedRunningTime="2026-11-04 02:49:26.409994987 +0000 UTC m=+1013.376496023" Nov 04 02:49:26.546569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.546474 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp"] Nov 04 02:49:26.550640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.550616 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.608866 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.608833 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp"] Nov 04 02:49:26.644627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.644589 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk4r\" (UniqueName: \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.644812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.644648 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.644812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.644724 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.745795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.745760 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk4r\" (UniqueName: \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.745998 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.745819 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.745998 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.745854 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.746558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.746527 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.748385 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.748359 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.808487 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.808417 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk4r\" (UniqueName: \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\") pod \"maas-api-revoke-sub-bc4db5d8-psnlp\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:26.863853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:26.863817 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:27.242460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.242432 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp"] Nov 04 02:49:27.242862 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:27.242837 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6eb948_dd47_434e_9000_e32177d3903a.slice/crio-2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e WatchSource:0}: Error finding container 2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e: Status 404 returned error can't find the container with id 2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e Nov 04 02:49:27.300367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.300322 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" event={"ID":"bd6eb948-dd47-434e-9000-e32177d3903a","Type":"ContainerStarted","Data":"2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e"} Nov 04 02:49:27.302073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.302045 4784 generic.go:356] "Generic (PLEG): container finished" podID="87f793cc-05d3-428b-a4a2-d36eb9950f42" containerID="cb4c1add07fd0945645ab540e4595513da61ecb45e28735bf8edadb6d93e2cd9" exitCode=0 Nov 04 02:49:27.302240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.302124 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" event={"ID":"87f793cc-05d3-428b-a4a2-d36eb9950f42","Type":"ContainerDied","Data":"cb4c1add07fd0945645ab540e4595513da61ecb45e28735bf8edadb6d93e2cd9"} Nov 04 02:49:27.500866 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.500829 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:27.657411 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.657373 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\" (UniqueName: \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\") pod \"2e579e29-8f23-432a-9ea2-6232f656c1f6\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " Nov 04 02:49:27.657608 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.657503 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\") pod \"2e579e29-8f23-432a-9ea2-6232f656c1f6\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " Nov 04 02:49:27.657608 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.657538 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\") pod \"2e579e29-8f23-432a-9ea2-6232f656c1f6\" (UID: \"2e579e29-8f23-432a-9ea2-6232f656c1f6\") " Nov 04 02:49:27.658306 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.658273 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca" pod "2e579e29-8f23-432a-9ea2-6232f656c1f6" (UID: "2e579e29-8f23-432a-9ea2-6232f656c1f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:27.660262 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.660232 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x" pod "2e579e29-8f23-432a-9ea2-6232f656c1f6" (UID: "2e579e29-8f23-432a-9ea2-6232f656c1f6"). InnerVolumeSpecName "kube-api-access-wch9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:27.660817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.660789 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca" pod "2e579e29-8f23-432a-9ea2-6232f656c1f6" (UID: "2e579e29-8f23-432a-9ea2-6232f656c1f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:27.758888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.758794 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wch9x\" (UniqueName: \"kubernetes.io/projected/2e579e29-8f23-432a-9ea2-6232f656c1f6-kube-api-access-wch9x\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:27.758888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.758829 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:27.758888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:27.758841 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e579e29-8f23-432a-9ea2-6232f656c1f6-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:28.312509 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.312466 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" event={"ID":"bd6eb948-dd47-434e-9000-e32177d3903a","Type":"ContainerStarted","Data":"dd65eac7ba1b965aa9edf5792292834c502afb06cd431abd894ef708466ba4ec"} Nov 04 02:49:28.315753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.315726 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" Nov 04 02:49:28.315924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.315757 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2" event={"ID":"2e579e29-8f23-432a-9ea2-6232f656c1f6","Type":"ContainerDied","Data":"f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841"} Nov 04 02:49:28.315924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.315798 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5f0feb0b116a393814a898c04c31c0bf137c6d15060958d9b884b2d21043841" Nov 04 02:49:28.439527 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.438539 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" podStartSLOduration=2.438517602 podStartE2EDuration="2.438517602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:28.436546542 +0000 UTC m=+1015.403047535" watchObservedRunningTime="2026-11-04 02:49:28.438517602 +0000 UTC m=+1015.405018596" Nov 04 02:49:28.700421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.695835 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:28.797601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.797558 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\") pod \"87f793cc-05d3-428b-a4a2-d36eb9950f42\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " Nov 04 02:49:28.798300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.798273 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\" (UniqueName: \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\") pod \"87f793cc-05d3-428b-a4a2-d36eb9950f42\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " Nov 04 02:49:28.798752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.798728 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\") pod \"87f793cc-05d3-428b-a4a2-d36eb9950f42\" (UID: \"87f793cc-05d3-428b-a4a2-d36eb9950f42\") " Nov 04 02:49:28.805426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.805382 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca" pod "87f793cc-05d3-428b-a4a2-d36eb9950f42" (UID: "87f793cc-05d3-428b-a4a2-d36eb9950f42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:28.827240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.823175 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca" pod "87f793cc-05d3-428b-a4a2-d36eb9950f42" (UID: "87f793cc-05d3-428b-a4a2-d36eb9950f42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:28.827240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.823383 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg" pod "87f793cc-05d3-428b-a4a2-d36eb9950f42" (UID: "87f793cc-05d3-428b-a4a2-d36eb9950f42"). InnerVolumeSpecName "kube-api-access-cxxvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:28.904218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.903154 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:28.904218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.903266 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f793cc-05d3-428b-a4a2-d36eb9950f42-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:28.904218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:28.903285 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxvg\" (UniqueName: \"kubernetes.io/projected/87f793cc-05d3-428b-a4a2-d36eb9950f42-kube-api-access-cxxvg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:29.335222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.335124 4784 generic.go:356] "Generic (PLEG): container finished" podID="bd6eb948-dd47-434e-9000-e32177d3903a" containerID="dd65eac7ba1b965aa9edf5792292834c502afb06cd431abd894ef708466ba4ec" exitCode=0 Nov 04 02:49:29.335772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.335281 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" event={"ID":"bd6eb948-dd47-434e-9000-e32177d3903a","Type":"ContainerDied","Data":"dd65eac7ba1b965aa9edf5792292834c502afb06cd431abd894ef708466ba4ec"} Nov 04 02:49:29.338224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.337880 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" event={"ID":"87f793cc-05d3-428b-a4a2-d36eb9950f42","Type":"ContainerDied","Data":"2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741"} Nov 04 02:49:29.338224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.337934 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fefb4d94f26962dbfe8fc3f126d8055ff449f7849778075cbade8deb5bda741" Nov 04 02:49:29.338224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.338007 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm" Nov 04 02:49:29.855528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.855491 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67"] Nov 04 02:49:29.856081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856068 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2e579e29-8f23-432a-9ea2-6232f656c1f6" containerName="revoke-keys" Nov 04 02:49:29.856134 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856082 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e579e29-8f23-432a-9ea2-6232f656c1f6" containerName="revoke-keys" Nov 04 02:49:29.856134 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856092 4784 cpu_manager.go:414] "containerMap: removing container" podUID="87f793cc-05d3-428b-a4a2-d36eb9950f42" containerName="revoke-keys" Nov 04 02:49:29.856134 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856097 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f793cc-05d3-428b-a4a2-d36eb9950f42" containerName="revoke-keys" Nov 04 02:49:29.856269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856201 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e579e29-8f23-432a-9ea2-6232f656c1f6" containerName="revoke-keys" Nov 04 02:49:29.856269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.856213 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f793cc-05d3-428b-a4a2-d36eb9950f42" containerName="revoke-keys" Nov 04 02:49:29.859894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.859864 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:29.916225 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.916171 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67"] Nov 04 02:49:29.928498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.928461 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8th7\" (UniqueName: \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:29.928711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.928683 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:29.928815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:29.928799 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.029791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.029752 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.029987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.029821 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.029987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.029905 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8th7\" (UniqueName: \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.030794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.030762 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.032517 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.032494 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.092804 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.092769 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8th7\" (UniqueName: \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\") pod \"maas-api-revoke-sub-3df2e539-pkm67\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.174970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.174925 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:30.530599 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.530570 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:30.560874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.560838 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67"] Nov 04 02:49:30.561333 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:30.561305 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c3f1d_b725_455a_93df_e86110e4a3f6.slice/crio-03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c WatchSource:0}: Error finding container 03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c: Status 404 returned error can't find the container with id 03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c Nov 04 02:49:30.638506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.638474 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\") pod \"bd6eb948-dd47-434e-9000-e32177d3903a\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " Nov 04 02:49:30.638696 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.638594 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\") pod \"bd6eb948-dd47-434e-9000-e32177d3903a\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " Nov 04 02:49:30.638696 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.638633 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\" (UniqueName: \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\") pod \"bd6eb948-dd47-434e-9000-e32177d3903a\" (UID: \"bd6eb948-dd47-434e-9000-e32177d3903a\") " Nov 04 02:49:30.638915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.638890 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca" pod "bd6eb948-dd47-434e-9000-e32177d3903a" (UID: "bd6eb948-dd47-434e-9000-e32177d3903a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:30.639105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.639087 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:30.640911 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.640884 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r" pod "bd6eb948-dd47-434e-9000-e32177d3903a" (UID: "bd6eb948-dd47-434e-9000-e32177d3903a"). InnerVolumeSpecName "kube-api-access-9wk4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:30.641003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.640982 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca" pod "bd6eb948-dd47-434e-9000-e32177d3903a" (UID: "bd6eb948-dd47-434e-9000-e32177d3903a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:30.740495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.740458 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6eb948-dd47-434e-9000-e32177d3903a-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:30.740495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:30.740488 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wk4r\" (UniqueName: \"kubernetes.io/projected/bd6eb948-dd47-434e-9000-e32177d3903a-kube-api-access-9wk4r\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:31.356315 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.356278 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" event={"ID":"ac6c3f1d-b725-455a-93df-e86110e4a3f6","Type":"ContainerStarted","Data":"a98224efe92316b9772e60677d57204dcdf80f2e21ca763d72a11b07d6ce4b46"} Nov 04 02:49:31.356641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.356619 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" event={"ID":"ac6c3f1d-b725-455a-93df-e86110e4a3f6","Type":"ContainerStarted","Data":"03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c"} Nov 04 02:49:31.358295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.358268 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" event={"ID":"bd6eb948-dd47-434e-9000-e32177d3903a","Type":"ContainerDied","Data":"2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e"} Nov 04 02:49:31.358424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.358309 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5d54b5185ceff3a7b5880d1af11daf9f39af32ed71e15bdd6225305127144e" Nov 04 02:49:31.358424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.358318 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp" Nov 04 02:49:31.526920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:31.526872 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" podStartSLOduration=2.526854272 podStartE2EDuration="2.526854272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:31.523982592 +0000 UTC m=+1018.490483584" watchObservedRunningTime="2026-11-04 02:49:31.526854272 +0000 UTC m=+1018.493355264" Nov 04 02:49:32.367413 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:32.367371 4784 generic.go:356] "Generic (PLEG): container finished" podID="ac6c3f1d-b725-455a-93df-e86110e4a3f6" containerID="a98224efe92316b9772e60677d57204dcdf80f2e21ca763d72a11b07d6ce4b46" exitCode=0 Nov 04 02:49:32.367927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:32.367461 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" event={"ID":"ac6c3f1d-b725-455a-93df-e86110e4a3f6","Type":"ContainerDied","Data":"a98224efe92316b9772e60677d57204dcdf80f2e21ca763d72a11b07d6ce4b46"} Nov 04 02:49:33.532285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.532243 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:33.676528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.676492 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\") pod \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " Nov 04 02:49:33.676707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.676543 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\") pod \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " Nov 04 02:49:33.676707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.676585 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\" (UniqueName: \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\") pod \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\" (UID: \"ac6c3f1d-b725-455a-93df-e86110e4a3f6\") " Nov 04 02:49:33.676991 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.676964 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca" pod "ac6c3f1d-b725-455a-93df-e86110e4a3f6" (UID: "ac6c3f1d-b725-455a-93df-e86110e4a3f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:33.677101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.677083 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:33.678908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.678871 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca" pod "ac6c3f1d-b725-455a-93df-e86110e4a3f6" (UID: "ac6c3f1d-b725-455a-93df-e86110e4a3f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:33.679033 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.679000 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7" pod "ac6c3f1d-b725-455a-93df-e86110e4a3f6" (UID: "ac6c3f1d-b725-455a-93df-e86110e4a3f6"). InnerVolumeSpecName "kube-api-access-t8th7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:33.777897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.777864 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c3f1d-b725-455a-93df-e86110e4a3f6-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:33.777897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:33.777896 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8th7\" (UniqueName: \"kubernetes.io/projected/ac6c3f1d-b725-455a-93df-e86110e4a3f6-kube-api-access-t8th7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:34.382224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:34.382181 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" Nov 04 02:49:34.382430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:34.382232 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67" event={"ID":"ac6c3f1d-b725-455a-93df-e86110e4a3f6","Type":"ContainerDied","Data":"03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c"} Nov 04 02:49:34.382430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:34.382292 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a740905f9ffcbeea74f3b922984a8f02cbfc2ea235f0c3f873e423a64c620c" Nov 04 02:49:36.401786 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:49:36.401748 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:49:36.603433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.603399 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-44dgj-nwd8w"] Nov 04 02:49:36.604022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604006 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c3f1d-b725-455a-93df-e86110e4a3f6" containerName="revoke-keys" Nov 04 02:49:36.604022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604023 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c3f1d-b725-455a-93df-e86110e4a3f6" containerName="revoke-keys" Nov 04 02:49:36.604147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604069 4784 cpu_manager.go:414] "containerMap: removing container" podUID="bd6eb948-dd47-434e-9000-e32177d3903a" containerName="revoke-keys" Nov 04 02:49:36.604147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604076 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6eb948-dd47-434e-9000-e32177d3903a" containerName="revoke-keys" Nov 04 02:49:36.604279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604176 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c3f1d-b725-455a-93df-e86110e4a3f6" containerName="revoke-keys" Nov 04 02:49:36.604279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.604200 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6eb948-dd47-434e-9000-e32177d3903a" containerName="revoke-keys" Nov 04 02:49:36.607862 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.607839 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.661711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.661618 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7xc9\"" Nov 04 02:49:36.664133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.664106 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-44dgj-nwd8w"] Nov 04 02:49:36.709378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.709341 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.709563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.709388 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.709563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.709540 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwgx\" (UniqueName: \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.709631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.709592 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.810982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.810945 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.810982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.810984 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.811231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.811055 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwgx\" (UniqueName: \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.811231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.811109 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.811513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.811490 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.811727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.811709 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.811836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.811818 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.873644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.873611 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwgx\" (UniqueName: \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\") pod \"periodic-gathering-44dgj-nwd8w\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:36.921544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:36.921512 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:49:37.303071 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:37.303023 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd416426e_a063_465e_836c_e6526d784642.slice/crio-80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06 WatchSource:0}: Error finding container 80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06: Status 404 returned error can't find the container with id 80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06 Nov 04 02:49:37.303462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:37.303435 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-44dgj-nwd8w"] Nov 04 02:49:37.408241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:37.408204 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" event={"ID":"d416426e-a063-465e-836c-e6526d784642","Type":"ContainerStarted","Data":"466e517b2c44ea743b14fb48f700c4a9dc1f1bdee8f92785390d980509318d5f"} Nov 04 02:49:37.408665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:37.408264 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" event={"ID":"d416426e-a063-465e-836c-e6526d784642","Type":"ContainerStarted","Data":"80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06"} Nov 04 02:49:37.529622 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:37.529565 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" podStartSLOduration=1.5295500789999998 podStartE2EDuration="1.529550079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:37.52761873 +0000 UTC m=+1024.494119725" watchObservedRunningTime="2026-11-04 02:49:37.529550079 +0000 UTC m=+1024.496051097" Nov 04 02:49:42.909790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:42.909705 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6"] Nov 04 02:49:42.914104 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:42.914080 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:42.971734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:42.971693 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6"] Nov 04 02:49:43.018070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.018036 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:49:43.018302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.018085 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:49:43.018302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.018102 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:49:43.018302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.018253 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:49:43.089765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.089725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgb9\" (UniqueName: \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.089945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.089797 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.090038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.090014 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.191122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.191079 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgb9\" (UniqueName: \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.191330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.191155 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.191396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.191378 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.192085 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.192057 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.193658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.193635 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.255648 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.255617 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgb9\" (UniqueName: \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\") pod \"maas-api-revoke-sub-a2aca13e-gdkw6\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.330479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.330435 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l"] Nov 04 02:49:43.340687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.340657 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.393116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.393075 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l"] Nov 04 02:49:43.494914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.494813 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.495091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.494972 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.495156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.495119 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdph\" (UniqueName: \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.528617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.528573 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:43.607785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.607746 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.607967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.607864 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.608042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.608012 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdph\" (UniqueName: \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.608946 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.608920 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.610464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.610425 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.670439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.670409 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdph\" (UniqueName: \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\") pod \"maas-api-revoke-sub-e199f9d2-fzb8l\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:43.908552 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:43.908518 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382f038a_26d0_4aff_8b61_c747cd78b678.slice/crio-be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef WatchSource:0}: Error finding container be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef: Status 404 returned error can't find the container with id be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef Nov 04 02:49:43.908552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.908523 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6"] Nov 04 02:49:43.956167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:43.956137 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:44.338263 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:44.338228 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff3ac3f_eb19_494d_ac25_b5ec506ca21c.slice/crio-8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4 WatchSource:0}: Error finding container 8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4: Status 404 returned error can't find the container with id 8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4 Nov 04 02:49:44.338846 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:44.338822 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l"] Nov 04 02:49:44.461076 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:44.460982 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" event={"ID":"aff3ac3f-eb19-494d-ac25-b5ec506ca21c","Type":"ContainerStarted","Data":"8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4"} Nov 04 02:49:44.462585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:44.462550 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" event={"ID":"382f038a-26d0-4aff-8b61-c747cd78b678","Type":"ContainerStarted","Data":"0dc1d8f91d3262816d920e2bc97ced38d7320213320b43c7fc4101c59da00213"} Nov 04 02:49:44.462739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:44.462599 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" event={"ID":"382f038a-26d0-4aff-8b61-c747cd78b678","Type":"ContainerStarted","Data":"be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef"} Nov 04 02:49:44.583667 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:44.583615 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" podStartSLOduration=2.583599453 podStartE2EDuration="2.583599453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:44.580235596 +0000 UTC m=+1031.546736601" watchObservedRunningTime="2026-11-04 02:49:44.583599453 +0000 UTC m=+1031.550100445" Nov 04 02:49:45.482129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.482095 4784 generic.go:356] "Generic (PLEG): container finished" podID="aff3ac3f-eb19-494d-ac25-b5ec506ca21c" containerID="b42efddce37cddb48ebc5fcfc2b009b0927e702322fcd28dc689566a59e2d271" exitCode=0 Nov 04 02:49:45.482638 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.482182 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" event={"ID":"aff3ac3f-eb19-494d-ac25-b5ec506ca21c","Type":"ContainerDied","Data":"b42efddce37cddb48ebc5fcfc2b009b0927e702322fcd28dc689566a59e2d271"} Nov 04 02:49:45.483898 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.483875 4784 generic.go:356] "Generic (PLEG): container finished" podID="382f038a-26d0-4aff-8b61-c747cd78b678" containerID="0dc1d8f91d3262816d920e2bc97ced38d7320213320b43c7fc4101c59da00213" exitCode=0 Nov 04 02:49:45.483999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.483925 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" event={"ID":"382f038a-26d0-4aff-8b61-c747cd78b678","Type":"ContainerDied","Data":"0dc1d8f91d3262816d920e2bc97ced38d7320213320b43c7fc4101c59da00213"} Nov 04 02:49:45.729051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.729015 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r"] Nov 04 02:49:45.737952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.737923 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.793583 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.793551 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r"] Nov 04 02:49:45.833243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.833175 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.833389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.833336 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.833389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.833377 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nwc\" (UniqueName: \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.935228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.934761 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.935228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.934886 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.935228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.934933 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nwc\" (UniqueName: \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.936464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.935608 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.940284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.938512 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:45.997948 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:45.997875 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nwc\" (UniqueName: \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\") pod \"maas-api-revoke-sub-a797974f-xl99r\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:46.059579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.059543 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:46.411246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.411141 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d"] Nov 04 02:49:46.416720 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.416693 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.441885 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:46.441848 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2559ba_8c67_4037_9486_0e5040e4a636.slice/crio-fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c WatchSource:0}: Error finding container fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c: Status 404 returned error can't find the container with id fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c Nov 04 02:49:46.442020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.442002 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.442093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.442037 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.442158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.442134 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gmq\" (UniqueName: \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.442318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.442300 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r"] Nov 04 02:49:46.472930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.472900 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d"] Nov 04 02:49:46.502926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.502869 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" event={"ID":"1d2559ba-8c67-4037-9486-0e5040e4a636","Type":"ContainerStarted","Data":"fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c"} Nov 04 02:49:46.544018 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.543961 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.544252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.544043 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.544252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.544086 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gmq\" (UniqueName: \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.545532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.545503 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.547242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.547219 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.616512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.616478 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gmq\" (UniqueName: \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\") pod \"maas-api-revoke-sub-ad449bda-pxp5d\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.690101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.690056 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:46.712177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.712152 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:46.731602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.731569 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:46.749328 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749294 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\") pod \"382f038a-26d0-4aff-8b61-c747cd78b678\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " Nov 04 02:49:46.749468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749357 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\" (UniqueName: \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\") pod \"382f038a-26d0-4aff-8b61-c747cd78b678\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " Nov 04 02:49:46.749536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749474 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\") pod \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " Nov 04 02:49:46.749707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749686 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca" pod "382f038a-26d0-4aff-8b61-c747cd78b678" (UID: "382f038a-26d0-4aff-8b61-c747cd78b678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:46.749772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749715 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\" (UniqueName: \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\") pod \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " Nov 04 02:49:46.749772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749754 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\") pod \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\" (UID: \"aff3ac3f-eb19-494d-ac25-b5ec506ca21c\") " Nov 04 02:49:46.749924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749773 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca" pod "aff3ac3f-eb19-494d-ac25-b5ec506ca21c" (UID: "aff3ac3f-eb19-494d-ac25-b5ec506ca21c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:46.749924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.749830 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\") pod \"382f038a-26d0-4aff-8b61-c747cd78b678\" (UID: \"382f038a-26d0-4aff-8b61-c747cd78b678\") " Nov 04 02:49:46.750357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.750305 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:46.750357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.750326 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:46.752163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.752136 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9" pod "382f038a-26d0-4aff-8b61-c747cd78b678" (UID: "382f038a-26d0-4aff-8b61-c747cd78b678"). InnerVolumeSpecName "kube-api-access-prgb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:46.752384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.752365 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca" pod "aff3ac3f-eb19-494d-ac25-b5ec506ca21c" (UID: "aff3ac3f-eb19-494d-ac25-b5ec506ca21c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:46.752446 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.752406 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph" pod "aff3ac3f-eb19-494d-ac25-b5ec506ca21c" (UID: "aff3ac3f-eb19-494d-ac25-b5ec506ca21c"). InnerVolumeSpecName "kube-api-access-nmdph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:46.752643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.752614 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca" pod "382f038a-26d0-4aff-8b61-c747cd78b678" (UID: "382f038a-26d0-4aff-8b61-c747cd78b678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:46.851291 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.851247 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdph\" (UniqueName: \"kubernetes.io/projected/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-kube-api-access-nmdph\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:46.851291 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.851293 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff3ac3f-eb19-494d-ac25-b5ec506ca21c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:46.851557 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.851310 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382f038a-26d0-4aff-8b61-c747cd78b678-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:46.851557 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:46.851324 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgb9\" (UniqueName: \"kubernetes.io/projected/382f038a-26d0-4aff-8b61-c747cd78b678-kube-api-access-prgb9\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:47.120637 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:47.120603 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c631659_850f_47ce_9e88_52ad342c750e.slice/crio-e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391 WatchSource:0}: Error finding container e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391: Status 404 returned error can't find the container with id e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391 Nov 04 02:49:47.120980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.120942 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d"] Nov 04 02:49:47.512686 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.512653 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" event={"ID":"0c631659-850f-47ce-9e88-52ad342c750e","Type":"ContainerStarted","Data":"6063f7a5b14c1853c32b186a99b130efae8db8772cdd4621de751ad88fcb9b5c"} Nov 04 02:49:47.513226 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.512718 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" event={"ID":"0c631659-850f-47ce-9e88-52ad342c750e","Type":"ContainerStarted","Data":"e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391"} Nov 04 02:49:47.514197 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.514169 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" Nov 04 02:49:47.514335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.514168 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l" event={"ID":"aff3ac3f-eb19-494d-ac25-b5ec506ca21c","Type":"ContainerDied","Data":"8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4"} Nov 04 02:49:47.514335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.514328 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c46fd6d9f346001b89f6bc0a5db75648353269415fa2dbc5754ebe5faf648b4" Nov 04 02:49:47.515655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.515613 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" Nov 04 02:49:47.515791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.515664 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6" event={"ID":"382f038a-26d0-4aff-8b61-c747cd78b678","Type":"ContainerDied","Data":"be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef"} Nov 04 02:49:47.515791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.515704 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be354722b8730ad51537dc1650017f9ce7732a82dec992aa89fa37eb6e3bd6ef" Nov 04 02:49:47.517506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.517480 4784 generic.go:356] "Generic (PLEG): container finished" podID="1d2559ba-8c67-4037-9486-0e5040e4a636" containerID="4f2d32d43c50870332653ffe33a429c59a134c34e17695bfc2591b642068f6ec" exitCode=0 Nov 04 02:49:47.517607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.517535 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" event={"ID":"1d2559ba-8c67-4037-9486-0e5040e4a636","Type":"ContainerDied","Data":"4f2d32d43c50870332653ffe33a429c59a134c34e17695bfc2591b642068f6ec"} Nov 04 02:49:47.806073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:47.806020 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" podStartSLOduration=1.8060046060000001 podStartE2EDuration="1.806004606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:47.634135045 +0000 UTC m=+1034.600636037" watchObservedRunningTime="2026-11-04 02:49:47.806004606 +0000 UTC m=+1034.772505598" Nov 04 02:49:48.047516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.047479 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp"] Nov 04 02:49:48.048156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048136 4784 cpu_manager.go:414] "containerMap: removing container" podUID="382f038a-26d0-4aff-8b61-c747cd78b678" containerName="revoke-keys" Nov 04 02:49:48.048287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048157 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382f038a-26d0-4aff-8b61-c747cd78b678" containerName="revoke-keys" Nov 04 02:49:48.048287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048177 4784 cpu_manager.go:414] "containerMap: removing container" podUID="aff3ac3f-eb19-494d-ac25-b5ec506ca21c" containerName="revoke-keys" Nov 04 02:49:48.048287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048202 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff3ac3f-eb19-494d-ac25-b5ec506ca21c" containerName="revoke-keys" Nov 04 02:49:48.048459 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048356 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="382f038a-26d0-4aff-8b61-c747cd78b678" containerName="revoke-keys" Nov 04 02:49:48.048459 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.048370 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff3ac3f-eb19-494d-ac25-b5ec506ca21c" containerName="revoke-keys" Nov 04 02:49:48.051853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.051831 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.070207 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.070152 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcmj\" (UniqueName: \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.070346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.070215 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.070346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.070339 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.111556 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.111521 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp"] Nov 04 02:49:48.125237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.125201 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 02:49:48.130551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.130526 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171429 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171471 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171607 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171637 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fds6x\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171710 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171730 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171824 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcmj\" (UniqueName: \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.171992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171842 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.171992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171884 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.171992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.171912 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.172153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.172062 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.172153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.172117 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.172586 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.172563 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.174586 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.174564 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.184903 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.184873 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3b7476-openshift-default-dockercfg-t2888\"" Nov 04 02:49:48.189694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.189662 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 02:49:48.235428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.235396 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcmj\" (UniqueName: \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\") pod \"maas-api-revoke-sub-6f60ae9e-5prkp\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.237672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.237642 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf"] Nov 04 02:49:48.241931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.241911 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.273522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273486 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273544 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv86\" (UniqueName: \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.273709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273595 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273614 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273680 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273715 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fds6x\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273757 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273776 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273814 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273886 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.273926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273912 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.274255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.273952 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.274255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.274017 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.274255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.274148 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.274378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.274343 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.274378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.274368 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.274523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.274500 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.276176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.276150 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.276614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.276594 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.300847 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.300813 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf"] Nov 04 02:49:48.336838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.336764 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.336973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.336932 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fds6x\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.364565 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.364534 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:48.377090 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.377059 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.377270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.377174 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv86\" (UniqueName: \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.377374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.377356 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.377787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.377761 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.379847 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.379824 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.448425 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.448395 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:48.449981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.449956 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv86\" (UniqueName: \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\") pod \"maas-api-revoke-sub-0a7f9e0f-bk9kf\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.532258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.532223 4784 generic.go:356] "Generic (PLEG): container finished" podID="0c631659-850f-47ce-9e88-52ad342c750e" containerID="6063f7a5b14c1853c32b186a99b130efae8db8772cdd4621de751ad88fcb9b5c" exitCode=0 Nov 04 02:49:48.532731 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.532573 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" event={"ID":"0c631659-850f-47ce-9e88-52ad342c750e","Type":"ContainerDied","Data":"6063f7a5b14c1853c32b186a99b130efae8db8772cdd4621de751ad88fcb9b5c"} Nov 04 02:49:48.560649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.560361 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:48.688024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.687999 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:48.753843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.753816 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp"] Nov 04 02:49:48.754471 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:48.754410 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bbfdbc_9cbe_4858_8ff9_a6b69c3b832c.slice/crio-ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf WatchSource:0}: Error finding container ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf: Status 404 returned error can't find the container with id ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf Nov 04 02:49:48.782851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.782826 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\" (UniqueName: \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\") pod \"1d2559ba-8c67-4037-9486-0e5040e4a636\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " Nov 04 02:49:48.782983 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.782872 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\") pod \"1d2559ba-8c67-4037-9486-0e5040e4a636\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " Nov 04 02:49:48.782983 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.782906 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\") pod \"1d2559ba-8c67-4037-9486-0e5040e4a636\" (UID: \"1d2559ba-8c67-4037-9486-0e5040e4a636\") " Nov 04 02:49:48.783318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.783290 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca" pod "1d2559ba-8c67-4037-9486-0e5040e4a636" (UID: "1d2559ba-8c67-4037-9486-0e5040e4a636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:48.783516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.783499 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:48.785004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.784983 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc" pod "1d2559ba-8c67-4037-9486-0e5040e4a636" (UID: "1d2559ba-8c67-4037-9486-0e5040e4a636"). InnerVolumeSpecName "kube-api-access-v2nwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:48.785070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.785043 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca" pod "1d2559ba-8c67-4037-9486-0e5040e4a636" (UID: "1d2559ba-8c67-4037-9486-0e5040e4a636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:48.840158 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:48.839582 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f70fa7_388f_45e7_bcb4_4687882be424.slice/crio-2712dab8dc256238c228ac9cb83ead73cd8649629a4d73cdab2bfaaab2a6ce91 WatchSource:0}: Error finding container 2712dab8dc256238c228ac9cb83ead73cd8649629a4d73cdab2bfaaab2a6ce91: Status 404 returned error can't find the container with id 2712dab8dc256238c228ac9cb83ead73cd8649629a4d73cdab2bfaaab2a6ce91 Nov 04 02:49:48.840366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.840292 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 02:49:48.842682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.842639 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:49:48.842799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.842734 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:49:48.842799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.842784 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:49:48.884524 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.884494 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nwc\" (UniqueName: \"kubernetes.io/projected/1d2559ba-8c67-4037-9486-0e5040e4a636-kube-api-access-v2nwc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:48.884524 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.884522 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2559ba-8c67-4037-9486-0e5040e4a636-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:48.967756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:48.967726 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf"] Nov 04 02:49:48.972241 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:48.972208 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95341af_f780_44e8_86ed_97ec51dcd0ab.slice/crio-e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780 WatchSource:0}: Error finding container e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780: Status 404 returned error can't find the container with id e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780 Nov 04 02:49:49.546659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.546604 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" event={"ID":"95f70fa7-388f-45e7-bcb4-4687882be424","Type":"ContainerStarted","Data":"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785"} Nov 04 02:49:49.547179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.546674 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" event={"ID":"95f70fa7-388f-45e7-bcb4-4687882be424","Type":"ContainerStarted","Data":"2712dab8dc256238c228ac9cb83ead73cd8649629a4d73cdab2bfaaab2a6ce91"} Nov 04 02:49:49.548651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.548624 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" event={"ID":"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c","Type":"ContainerStarted","Data":"71634faf91118bbf7583fadf0f9b1dfc80ed8a35ee12b5dee006ab752a31a062"} Nov 04 02:49:49.548807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.548665 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" event={"ID":"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c","Type":"ContainerStarted","Data":"ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf"} Nov 04 02:49:49.550038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.550015 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" Nov 04 02:49:49.550038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.550022 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r" event={"ID":"1d2559ba-8c67-4037-9486-0e5040e4a636","Type":"ContainerDied","Data":"fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c"} Nov 04 02:49:49.550224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.550063 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa6fedd16267022d2b60f9344f97bed3815e963bc3d7279888d08019048f217c" Nov 04 02:49:49.551680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.551659 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" event={"ID":"e95341af-f780-44e8-86ed-97ec51dcd0ab","Type":"ContainerStarted","Data":"10266fd931261bfeddab0fb4799e47123d6de4b2a20433e308e1df5b8169ddac"} Nov 04 02:49:49.551783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.551706 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" event={"ID":"e95341af-f780-44e8-86ed-97ec51dcd0ab","Type":"ContainerStarted","Data":"e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780"} Nov 04 02:49:49.681212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.678729 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podStartSLOduration=1.678708999 podStartE2EDuration="1.678708999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:49.677466143 +0000 UTC m=+1036.643967146" watchObservedRunningTime="2026-11-04 02:49:49.678708999 +0000 UTC m=+1036.645209994" Nov 04 02:49:49.751751 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.751721 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:49.797714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.797680 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\") pod \"0c631659-850f-47ce-9e88-52ad342c750e\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " Nov 04 02:49:49.797906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.797779 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\" (UniqueName: \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\") pod \"0c631659-850f-47ce-9e88-52ad342c750e\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " Nov 04 02:49:49.797906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.797826 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\") pod \"0c631659-850f-47ce-9e88-52ad342c750e\" (UID: \"0c631659-850f-47ce-9e88-52ad342c750e\") " Nov 04 02:49:49.798241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.798218 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca" pod "0c631659-850f-47ce-9e88-52ad342c750e" (UID: "0c631659-850f-47ce-9e88-52ad342c750e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:49.798480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.798460 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:49.800345 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.800292 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" podStartSLOduration=1.800279295 podStartE2EDuration="1.800279295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:49.79730035 +0000 UTC m=+1036.763801343" watchObservedRunningTime="2026-11-04 02:49:49.800279295 +0000 UTC m=+1036.766780286" Nov 04 02:49:49.800498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.800476 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq" pod "0c631659-850f-47ce-9e88-52ad342c750e" (UID: "0c631659-850f-47ce-9e88-52ad342c750e"). InnerVolumeSpecName "kube-api-access-f7gmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:49.800607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.800584 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca" pod "0c631659-850f-47ce-9e88-52ad342c750e" (UID: "0c631659-850f-47ce-9e88-52ad342c750e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:49.899263 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.899226 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7gmq\" (UniqueName: \"kubernetes.io/projected/0c631659-850f-47ce-9e88-52ad342c750e-kube-api-access-f7gmq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:49.899263 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.899259 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c631659-850f-47ce-9e88-52ad342c750e-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:49.917454 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:49.917400 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" podStartSLOduration=1.9173857619999999 podStartE2EDuration="1.917385762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:49.914522535 +0000 UTC m=+1036.881023538" watchObservedRunningTime="2026-11-04 02:49:49.917385762 +0000 UTC m=+1036.883886801" Nov 04 02:49:50.449206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.449148 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:50.450875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.450852 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:50.450979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.450900 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:50.598212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.597127 4784 generic.go:356] "Generic (PLEG): container finished" podID="e95341af-f780-44e8-86ed-97ec51dcd0ab" containerID="10266fd931261bfeddab0fb4799e47123d6de4b2a20433e308e1df5b8169ddac" exitCode=0 Nov 04 02:49:50.598212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.597350 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" event={"ID":"e95341af-f780-44e8-86ed-97ec51dcd0ab","Type":"ContainerDied","Data":"10266fd931261bfeddab0fb4799e47123d6de4b2a20433e308e1df5b8169ddac"} Nov 04 02:49:50.604598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.604560 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" event={"ID":"0c631659-850f-47ce-9e88-52ad342c750e","Type":"ContainerDied","Data":"e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391"} Nov 04 02:49:50.604865 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.604849 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a169f1a7e6d94df499f8b6989e5305f91c8a28ac9d2f21c07723c855ad6391" Nov 04 02:49:50.605026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.605013 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d" Nov 04 02:49:50.693644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.693419 4784 generic.go:356] "Generic (PLEG): container finished" podID="88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" containerID="71634faf91118bbf7583fadf0f9b1dfc80ed8a35ee12b5dee006ab752a31a062" exitCode=0 Nov 04 02:49:50.693644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:50.693500 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" event={"ID":"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c","Type":"ContainerDied","Data":"71634faf91118bbf7583fadf0f9b1dfc80ed8a35ee12b5dee006ab752a31a062"} Nov 04 02:49:51.454028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:51.453992 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:51.454376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:51.454347 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:52.613576 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.598870 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:52.630569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.621133 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:52.791258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.779627 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:52.791868 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.788502 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" event={"ID":"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c","Type":"ContainerDied","Data":"ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf"} Nov 04 02:49:52.792113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.792094 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5adf4f74e1ace71c7afce1b585bae09241d0ce85cd0266285878d6710dbccf" Nov 04 02:49:52.846701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.846665 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\") pod \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " Nov 04 02:49:52.847603 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.847578 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\" (UniqueName: \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\") pod \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " Nov 04 02:49:52.847956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.847921 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\") pod \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\" (UID: \"88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c\") " Nov 04 02:49:52.848406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.848374 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca" pod "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" (UID: "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:52.848736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.848721 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:52.849388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.849341 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca" pod "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" (UID: "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:52.850302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.850281 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj" pod "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" (UID: "88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c"). InnerVolumeSpecName "kube-api-access-fzcmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:52.853486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.853464 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:52.949706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.949667 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\") pod \"e95341af-f780-44e8-86ed-97ec51dcd0ab\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " Nov 04 02:49:52.949910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.949796 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\" (UniqueName: \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\") pod \"e95341af-f780-44e8-86ed-97ec51dcd0ab\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " Nov 04 02:49:52.949910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.949822 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\") pod \"e95341af-f780-44e8-86ed-97ec51dcd0ab\" (UID: \"e95341af-f780-44e8-86ed-97ec51dcd0ab\") " Nov 04 02:49:52.950180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.950156 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcmj\" (UniqueName: \"kubernetes.io/projected/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-kube-api-access-fzcmj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:52.950180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.950175 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:52.950353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.950231 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca" pod "e95341af-f780-44e8-86ed-97ec51dcd0ab" (UID: "e95341af-f780-44e8-86ed-97ec51dcd0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:52.952012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.951991 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca" pod "e95341af-f780-44e8-86ed-97ec51dcd0ab" (UID: "e95341af-f780-44e8-86ed-97ec51dcd0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:52.952079 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:52.952030 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86" pod "e95341af-f780-44e8-86ed-97ec51dcd0ab" (UID: "e95341af-f780-44e8-86ed-97ec51dcd0ab"). InnerVolumeSpecName "kube-api-access-rxv86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:53.051296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.051254 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:53.051296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.051298 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv86\" (UniqueName: \"kubernetes.io/projected/e95341af-f780-44e8-86ed-97ec51dcd0ab-kube-api-access-rxv86\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:53.051519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.051314 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95341af-f780-44e8-86ed-97ec51dcd0ab-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:53.450523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.450482 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:53.450682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.450543 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:53.797703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.797672 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" Nov 04 02:49:53.797703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.797683 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf" event={"ID":"e95341af-f780-44e8-86ed-97ec51dcd0ab","Type":"ContainerDied","Data":"e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780"} Nov 04 02:49:53.798164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.797740 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6753efa08bb73dec43a080964b5dcb74aa137ead2f3606ae220f48102319780" Nov 04 02:49:53.798164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:53.797862 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp" Nov 04 02:49:54.111559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.111457 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5"] Nov 04 02:49:54.112054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112037 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e95341af-f780-44e8-86ed-97ec51dcd0ab" containerName="revoke-keys" Nov 04 02:49:54.112054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112052 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95341af-f780-44e8-86ed-97ec51dcd0ab" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112067 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0c631659-850f-47ce-9e88-52ad342c750e" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112073 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c631659-850f-47ce-9e88-52ad342c750e" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112082 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1d2559ba-8c67-4037-9486-0e5040e4a636" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112087 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2559ba-8c67-4037-9486-0e5040e4a636" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112103 4784 cpu_manager.go:414] "containerMap: removing container" podUID="88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112107 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112207 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2559ba-8c67-4037-9486-0e5040e4a636" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112218 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112231 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95341af-f780-44e8-86ed-97ec51dcd0ab" containerName="revoke-keys" Nov 04 02:49:54.112243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.112237 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c631659-850f-47ce-9e88-52ad342c750e" containerName="revoke-keys" Nov 04 02:49:54.115890 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.115869 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.178214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.175019 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:49:54.178214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.175129 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:49:54.178214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.175255 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:49:54.178214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.175317 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:49:54.186652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.186615 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5"] Nov 04 02:49:54.187346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.187318 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njh84\" (UniqueName: \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.187607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.187586 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.187702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.187650 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.290011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.289958 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.290234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.290017 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.290234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.290153 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njh84\" (UniqueName: \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.291246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.291204 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.293071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.293046 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.353170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.353139 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njh84\" (UniqueName: \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\") pod \"maas-api-revoke-sub-c7c37478-szlz5\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.430574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.430546 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:54.448965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.448937 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:54.449073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.448995 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:54.837229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:54.837180 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5"] Nov 04 02:49:54.837641 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:54.837508 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3613e1_a4ff_4a7d_81ce_417cc2def46e.slice/crio-ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462 WatchSource:0}: Error finding container ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462: Status 404 returned error can't find the container with id ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462 Nov 04 02:49:55.449353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:55.449311 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 02:49:55.449540 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:55.449381 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 02:49:55.815040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:55.815002 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" event={"ID":"2d3613e1-a4ff-4a7d-81ce-417cc2def46e","Type":"ContainerStarted","Data":"ca8ffd8c6b5f7034aed2161d076d0397a8063cdf0efb69bb6bafaccd9080879f"} Nov 04 02:49:55.815293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:55.815059 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" event={"ID":"2d3613e1-a4ff-4a7d-81ce-417cc2def46e","Type":"ContainerStarted","Data":"ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462"} Nov 04 02:49:55.950173 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:55.950105 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" podStartSLOduration=1.950083566 podStartE2EDuration="1.950083566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:55.944445727 +0000 UTC m=+1042.910946738" watchObservedRunningTime="2026-11-04 02:49:55.950083566 +0000 UTC m=+1042.916584558" Nov 04 02:49:56.600676 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.600641 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4"] Nov 04 02:49:56.614551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.614515 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.666474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.664152 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4"] Nov 04 02:49:56.720691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.720638 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.720691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.720691 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.720939 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.720878 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnhp\" (UniqueName: \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.824225 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.822862 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnhp\" (UniqueName: \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.824705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.824661 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.824852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.824837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.827258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.827229 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.828902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.828872 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.831519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.831490 4784 generic.go:356] "Generic (PLEG): container finished" podID="2d3613e1-a4ff-4a7d-81ce-417cc2def46e" containerID="ca8ffd8c6b5f7034aed2161d076d0397a8063cdf0efb69bb6bafaccd9080879f" exitCode=0 Nov 04 02:49:56.831605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.831573 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" event={"ID":"2d3613e1-a4ff-4a7d-81ce-417cc2def46e","Type":"ContainerDied","Data":"ca8ffd8c6b5f7034aed2161d076d0397a8063cdf0efb69bb6bafaccd9080879f"} Nov 04 02:49:56.887994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.887927 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnhp\" (UniqueName: \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\") pod \"maas-api-revoke-sub-0b86fb9a-rc2z4\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:56.926290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.924762 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:56.926290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.925983 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:56.930366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.930335 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 02:49:56.931441 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:56.931416 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:49:57.335783 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:49:57.335678 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdec99c5_31d3_408a_84e9_2b5a37490278.slice/crio-04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667 WatchSource:0}: Error finding container 04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667: Status 404 returned error can't find the container with id 04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667 Nov 04 02:49:57.336883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:57.336853 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4"] Nov 04 02:49:57.841059 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:57.841011 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" event={"ID":"fdec99c5-31d3-408a-84e9-2b5a37490278","Type":"ContainerStarted","Data":"64f857a0247a8cc81f026e898b4a3c03bd9c68968b2bba3543a7ce58dbeb8968"} Nov 04 02:49:57.841258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:57.841074 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" event={"ID":"fdec99c5-31d3-408a-84e9-2b5a37490278","Type":"ContainerStarted","Data":"04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667"} Nov 04 02:49:57.964965 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:57.964917 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" podStartSLOduration=1.964902361 podStartE2EDuration="1.964902361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:57.962963285 +0000 UTC m=+1044.929464277" watchObservedRunningTime="2026-11-04 02:49:57.964902361 +0000 UTC m=+1044.931403352" Nov 04 02:49:57.991876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:57.991848 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:49:58.043244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.043210 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\") pod \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " Nov 04 02:49:58.043453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.043291 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\") pod \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " Nov 04 02:49:58.043453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.043353 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\" (UniqueName: \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\") pod \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\" (UID: \"2d3613e1-a4ff-4a7d-81ce-417cc2def46e\") " Nov 04 02:49:58.043706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.043674 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca" pod "2d3613e1-a4ff-4a7d-81ce-417cc2def46e" (UID: "2d3613e1-a4ff-4a7d-81ce-417cc2def46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:58.044032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.043972 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:58.045553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.045528 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca" pod "2d3613e1-a4ff-4a7d-81ce-417cc2def46e" (UID: "2d3613e1-a4ff-4a7d-81ce-417cc2def46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:49:58.045661 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.045646 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84" pod "2d3613e1-a4ff-4a7d-81ce-417cc2def46e" (UID: "2d3613e1-a4ff-4a7d-81ce-417cc2def46e"). InnerVolumeSpecName "kube-api-access-njh84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:58.145013 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.144975 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njh84\" (UniqueName: \"kubernetes.io/projected/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-kube-api-access-njh84\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:58.145013 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.145017 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3613e1-a4ff-4a7d-81ce-417cc2def46e-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:58.850638 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.850598 4784 generic.go:356] "Generic (PLEG): container finished" podID="fdec99c5-31d3-408a-84e9-2b5a37490278" containerID="64f857a0247a8cc81f026e898b4a3c03bd9c68968b2bba3543a7ce58dbeb8968" exitCode=0 Nov 04 02:49:58.851081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.850676 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" event={"ID":"fdec99c5-31d3-408a-84e9-2b5a37490278","Type":"ContainerDied","Data":"64f857a0247a8cc81f026e898b4a3c03bd9c68968b2bba3543a7ce58dbeb8968"} Nov 04 02:49:58.852386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.852361 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" event={"ID":"2d3613e1-a4ff-4a7d-81ce-417cc2def46e","Type":"ContainerDied","Data":"ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462"} Nov 04 02:49:58.852508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.852413 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3ee35c6484fb88be621245f9be6f09f5664affc95e70a10c19ed150513a462" Nov 04 02:49:58.852508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:49:58.852368 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5" Nov 04 02:50:00.026947 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.026919 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:50:00.065916 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.065866 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\" (UniqueName: \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\") pod \"fdec99c5-31d3-408a-84e9-2b5a37490278\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " Nov 04 02:50:00.066112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.065981 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\") pod \"fdec99c5-31d3-408a-84e9-2b5a37490278\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " Nov 04 02:50:00.066112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.066051 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\") pod \"fdec99c5-31d3-408a-84e9-2b5a37490278\" (UID: \"fdec99c5-31d3-408a-84e9-2b5a37490278\") " Nov 04 02:50:00.066435 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.066401 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca" pod "fdec99c5-31d3-408a-84e9-2b5a37490278" (UID: "fdec99c5-31d3-408a-84e9-2b5a37490278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:50:00.068425 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.068399 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp" pod "fdec99c5-31d3-408a-84e9-2b5a37490278" (UID: "fdec99c5-31d3-408a-84e9-2b5a37490278"). InnerVolumeSpecName "kube-api-access-rlnhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:00.068540 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.068439 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca" pod "fdec99c5-31d3-408a-84e9-2b5a37490278" (UID: "fdec99c5-31d3-408a-84e9-2b5a37490278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:50:00.168071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.168034 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnhp\" (UniqueName: \"kubernetes.io/projected/fdec99c5-31d3-408a-84e9-2b5a37490278-kube-api-access-rlnhp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:00.168253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.168076 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:00.168253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.168093 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdec99c5-31d3-408a-84e9-2b5a37490278-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:00.868232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.868201 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" Nov 04 02:50:00.868444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.868229 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4" event={"ID":"fdec99c5-31d3-408a-84e9-2b5a37490278","Type":"ContainerDied","Data":"04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667"} Nov 04 02:50:00.868444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:00.868285 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04186fba466f75c75c5f24c38c4c5cee1970ee8949995c98274ec5a6060bd667" Nov 04 02:50:01.890174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890137 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc"] Nov 04 02:50:01.890773 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890758 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2d3613e1-a4ff-4a7d-81ce-417cc2def46e" containerName="revoke-keys" Nov 04 02:50:01.890821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890775 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3613e1-a4ff-4a7d-81ce-417cc2def46e" containerName="revoke-keys" Nov 04 02:50:01.890821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890796 4784 cpu_manager.go:414] "containerMap: removing container" podUID="fdec99c5-31d3-408a-84e9-2b5a37490278" containerName="revoke-keys" Nov 04 02:50:01.890821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890801 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdec99c5-31d3-408a-84e9-2b5a37490278" containerName="revoke-keys" Nov 04 02:50:01.890918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890903 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdec99c5-31d3-408a-84e9-2b5a37490278" containerName="revoke-keys" Nov 04 02:50:01.890952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.890926 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3613e1-a4ff-4a7d-81ce-417cc2def46e" containerName="revoke-keys" Nov 04 02:50:01.894746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.894728 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:01.954250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.954216 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc"] Nov 04 02:50:01.988384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.988348 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:50:01.988792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.988768 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:01.988908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.988799 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjtdt\" (UniqueName: \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:01.988908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.988816 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:01.999746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.999714 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:50:01.999871 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.999767 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:50:01.999871 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:01.999818 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:50:02.007241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.007201 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 02:50:02.011561 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.011544 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.072815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.070715 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 02:50:02.090298 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090267 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090381 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090401 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090545 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfmm\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090618 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090605 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.090654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090629 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjtdt\" (UniqueName: \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.090654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090649 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.090729 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090705 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090731 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090785 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090756 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.090922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090897 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.091000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.090953 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.091555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.091495 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.093423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.093397 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.153241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.153152 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjtdt\" (UniqueName: \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\") pod \"maas-api-revoke-sub-aaffa84c-8c6sc\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.191829 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.191793 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.191829 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.191828 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.191847 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.191977 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192016 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192082 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192179 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192254 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192321 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192348 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfmm\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192769 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192800 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192966 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192880 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.192966 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.192888 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.194356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.194336 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.194869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.194847 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.208825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.208803 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:02.255340 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.255311 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.256010 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.255992 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfmm\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.327156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.327126 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:02.592629 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:02.592598 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3735aa5e_9240_4e83_a052_e6b90826d98e.slice/crio-0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e WatchSource:0}: Error finding container 0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e: Status 404 returned error can't find the container with id 0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e Nov 04 02:50:02.593394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.593370 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc"] Nov 04 02:50:02.716404 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:02.716372 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cb22f0_97b4_42af_8f2e_6adc4b331bdb.slice/crio-cdc769e51ef83c541fbd2eccd42f7538ec9159b42df4683b1ec7948650fd9cf7 WatchSource:0}: Error finding container cdc769e51ef83c541fbd2eccd42f7538ec9159b42df4683b1ec7948650fd9cf7: Status 404 returned error can't find the container with id cdc769e51ef83c541fbd2eccd42f7538ec9159b42df4683b1ec7948650fd9cf7 Nov 04 02:50:02.716674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.716650 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 02:50:02.719009 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.718978 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:02.719095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.719041 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:02.719095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.719070 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:02.886655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.886551 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" event={"ID":"24cb22f0-97b4-42af-8f2e-6adc4b331bdb","Type":"ContainerStarted","Data":"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2"} Nov 04 02:50:02.886655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.886648 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" event={"ID":"24cb22f0-97b4-42af-8f2e-6adc4b331bdb","Type":"ContainerStarted","Data":"cdc769e51ef83c541fbd2eccd42f7538ec9159b42df4683b1ec7948650fd9cf7"} Nov 04 02:50:02.888711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:02.888676 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" event={"ID":"3735aa5e-9240-4e83-a052-e6b90826d98e","Type":"ContainerStarted","Data":"0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e"} Nov 04 02:50:03.014802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.014736 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podStartSLOduration=2.014717443 podStartE2EDuration="2.014717443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:03.008967698 +0000 UTC m=+1049.975468693" watchObservedRunningTime="2026-11-04 02:50:03.014717443 +0000 UTC m=+1049.981218438" Nov 04 02:50:03.327617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.327586 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:03.329406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.329381 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:03.329533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.329424 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:03.897482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.897449 4784 generic.go:356] "Generic (PLEG): container finished" podID="3735aa5e-9240-4e83-a052-e6b90826d98e" containerID="8c0b18f38cc0c983555cdbb6a758817ae6c3e953d1c7221aa35bcffd79d6ff2f" exitCode=0 Nov 04 02:50:03.897691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:03.897535 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" event={"ID":"3735aa5e-9240-4e83-a052-e6b90826d98e","Type":"ContainerDied","Data":"8c0b18f38cc0c983555cdbb6a758817ae6c3e953d1c7221aa35bcffd79d6ff2f"} Nov 04 02:50:04.327945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:04.327906 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:04.328470 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:04.327969 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:05.124751 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.124724 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:05.228630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.228591 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\" (UniqueName: \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\") pod \"3735aa5e-9240-4e83-a052-e6b90826d98e\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " Nov 04 02:50:05.228993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.228756 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\") pod \"3735aa5e-9240-4e83-a052-e6b90826d98e\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " Nov 04 02:50:05.228993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.228945 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\") pod \"3735aa5e-9240-4e83-a052-e6b90826d98e\" (UID: \"3735aa5e-9240-4e83-a052-e6b90826d98e\") " Nov 04 02:50:05.229299 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.229271 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca" pod "3735aa5e-9240-4e83-a052-e6b90826d98e" (UID: "3735aa5e-9240-4e83-a052-e6b90826d98e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:50:05.229453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.229435 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:05.231570 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.231541 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca" pod "3735aa5e-9240-4e83-a052-e6b90826d98e" (UID: "3735aa5e-9240-4e83-a052-e6b90826d98e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:50:05.231701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.231565 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt" pod "3735aa5e-9240-4e83-a052-e6b90826d98e" (UID: "3735aa5e-9240-4e83-a052-e6b90826d98e"). InnerVolumeSpecName "kube-api-access-sjtdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:05.330214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.329767 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:05.330214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.329838 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:05.331037 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.330624 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3735aa5e-9240-4e83-a052-e6b90826d98e-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:05.331037 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.330654 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjtdt\" (UniqueName: \"kubernetes.io/projected/3735aa5e-9240-4e83-a052-e6b90826d98e-kube-api-access-sjtdt\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:05.915932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.915686 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" Nov 04 02:50:05.915932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.915683 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc" event={"ID":"3735aa5e-9240-4e83-a052-e6b90826d98e","Type":"ContainerDied","Data":"0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e"} Nov 04 02:50:05.915932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:05.915847 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9160552ed4ae6611f148414a9b93bdda73e204fae91e4aef1e20f33e4f8a8e" Nov 04 02:50:06.333853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:06.329816 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:06.333853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:06.329883 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:06.441257 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:50:06.436899 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:50:07.328056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:07.328013 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:07.328438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:07.328399 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:08.328421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:08.327801 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:08.328421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:08.327881 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:09.331580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:09.329587 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 02:50:09.331580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:09.329652 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 02:50:10.740765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:10.740726 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:10.741672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:10.741649 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:10.742341 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:10.742318 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 02:50:11.086348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.086270 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf"] Nov 04 02:50:11.087275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.087251 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3735aa5e-9240-4e83-a052-e6b90826d98e" containerName="revoke-keys" Nov 04 02:50:11.087442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.087425 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3735aa5e-9240-4e83-a052-e6b90826d98e" containerName="revoke-keys" Nov 04 02:50:11.087747 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.087730 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3735aa5e-9240-4e83-a052-e6b90826d98e" containerName="revoke-keys" Nov 04 02:50:11.093283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.093266 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.173691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.173651 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:50:11.180271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.174042 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:50:11.180271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.176314 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:50:11.180271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.176515 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:50:11.209066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.209032 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf"] Nov 04 02:50:11.215043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.214996 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.215254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.215100 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr9b\" (UniqueName: \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.215254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.215241 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.316394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.316299 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr9b\" (UniqueName: \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.316623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.316487 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.316707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.316632 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.317824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.317789 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.320340 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.320304 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.379352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.379258 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr9b\" (UniqueName: \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\") pod \"maas-api-revoke-sub-e06f7523-8klrf\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.422230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.422171 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:11.816210 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:11.816163 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf"] Nov 04 02:50:11.816657 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:11.816395 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e757a7_15ab_48e8_a173_f5860fbaa43b.slice/crio-d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d WatchSource:0}: Error finding container d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d: Status 404 returned error can't find the container with id d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d Nov 04 02:50:12.083626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:12.083590 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" event={"ID":"54e757a7-15ab-48e8-a173-f5860fbaa43b","Type":"ContainerStarted","Data":"d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d"} Nov 04 02:50:13.093252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:13.093160 4784 generic.go:356] "Generic (PLEG): container finished" podID="54e757a7-15ab-48e8-a173-f5860fbaa43b" containerID="e7eb82df59730d49da66899ad492406084e22ef98e51a03e7013f9ed584eb5d3" exitCode=0 Nov 04 02:50:13.093252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:13.093224 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" event={"ID":"54e757a7-15ab-48e8-a173-f5860fbaa43b","Type":"ContainerDied","Data":"e7eb82df59730d49da66899ad492406084e22ef98e51a03e7013f9ed584eb5d3"} Nov 04 02:50:14.255862 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.255837 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:14.354553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.354446 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\") pod \"54e757a7-15ab-48e8-a173-f5860fbaa43b\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " Nov 04 02:50:14.354553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.354503 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\" (UniqueName: \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\") pod \"54e757a7-15ab-48e8-a173-f5860fbaa43b\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " Nov 04 02:50:14.354775 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.354690 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\") pod \"54e757a7-15ab-48e8-a173-f5860fbaa43b\" (UID: \"54e757a7-15ab-48e8-a173-f5860fbaa43b\") " Nov 04 02:50:14.355100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.355058 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca" pod "54e757a7-15ab-48e8-a173-f5860fbaa43b" (UID: "54e757a7-15ab-48e8-a173-f5860fbaa43b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:50:14.355402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.355168 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:14.356977 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.356938 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b" pod "54e757a7-15ab-48e8-a173-f5860fbaa43b" (UID: "54e757a7-15ab-48e8-a173-f5860fbaa43b"). InnerVolumeSpecName "kube-api-access-fmr9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:14.357097 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.357034 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca" pod "54e757a7-15ab-48e8-a173-f5860fbaa43b" (UID: "54e757a7-15ab-48e8-a173-f5860fbaa43b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:50:14.456204 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.456151 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e757a7-15ab-48e8-a173-f5860fbaa43b-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:14.456393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:14.456181 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmr9b\" (UniqueName: \"kubernetes.io/projected/54e757a7-15ab-48e8-a173-f5860fbaa43b-kube-api-access-fmr9b\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:15.109144 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:15.109113 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" Nov 04 02:50:15.109315 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:15.109117 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf" event={"ID":"54e757a7-15ab-48e8-a173-f5860fbaa43b","Type":"ContainerDied","Data":"d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d"} Nov 04 02:50:15.109315 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:15.109272 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d698d3e4e9a672fd847f7c9600a9bce7502ada3cdf63c48938c6c8e1531b512d" Nov 04 02:50:16.824896 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.824855 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 02:50:16.825472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.825458 4784 cpu_manager.go:414] "containerMap: removing container" podUID="54e757a7-15ab-48e8-a173-f5860fbaa43b" containerName="revoke-keys" Nov 04 02:50:16.825516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.825473 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e757a7-15ab-48e8-a173-f5860fbaa43b" containerName="revoke-keys" Nov 04 02:50:16.825607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.825596 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e757a7-15ab-48e8-a173-f5860fbaa43b" containerName="revoke-keys" Nov 04 02:50:16.829440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.829424 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.885705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.885664 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.885868 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.885817 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.885929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.885888 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.885977 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.885965 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.886026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.885998 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrx7\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.886113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.886095 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.886175 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.886140 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.886255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.886216 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.886290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.886269 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.933082 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.933048 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 02:50:16.988145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988097 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988239 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988272 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988371 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988416 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrx7\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988455 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988493 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988520 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988591 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.988790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988774 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.989024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.988997 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.989438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.989409 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.989534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.989420 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.989825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.989801 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.991642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.991594 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:16.991830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:16.991807 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:17.054116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.054083 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrx7\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:17.056806 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.056776 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:17.143598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.143421 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:17.362507 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.362466 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t"] Nov 04 02:50:17.368060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.368033 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.422552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.422522 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:50:17.422724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.422618 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:50:17.422724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.422704 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:50:17.422807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.422745 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:50:17.425731 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.425703 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t"] Nov 04 02:50:17.496029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.495991 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.496182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.496045 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.496182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.496166 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8mv\" (UniqueName: \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.597135 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.597092 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.597332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.597145 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.597332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.597233 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8mv\" (UniqueName: \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.597942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.597879 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.599740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.599720 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.610838 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:17.610811 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95642dea_64ed_401d_893f_d1def7ecee3e.slice/crio-a51babf2ecc13b45d833e2462bc5addfcf8a8fd3c41a93b5862f1223ed85d71a WatchSource:0}: Error finding container a51babf2ecc13b45d833e2462bc5addfcf8a8fd3c41a93b5862f1223ed85d71a: Status 404 returned error can't find the container with id a51babf2ecc13b45d833e2462bc5addfcf8a8fd3c41a93b5862f1223ed85d71a Nov 04 02:50:17.611103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.611081 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 02:50:17.613126 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.613093 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:17.613259 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.613174 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:17.613259 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.613249 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:17.660534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.660504 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8mv\" (UniqueName: \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\") pod \"maas-api-revoke-sub-dc247602-l2h5t\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:17.683706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:17.683679 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:18.075153 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:18.075114 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fedc871_3ff0_4689_a2e3_29a534cc92dd.slice/crio-190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e WatchSource:0}: Error finding container 190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e: Status 404 returned error can't find the container with id 190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e Nov 04 02:50:18.076001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:18.075968 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t"] Nov 04 02:50:18.144147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:18.144097 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" event={"ID":"95642dea-64ed-401d-893f-d1def7ecee3e","Type":"ContainerStarted","Data":"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b"} Nov 04 02:50:18.144365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:18.144166 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" event={"ID":"95642dea-64ed-401d-893f-d1def7ecee3e","Type":"ContainerStarted","Data":"a51babf2ecc13b45d833e2462bc5addfcf8a8fd3c41a93b5862f1223ed85d71a"} Nov 04 02:50:18.146034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:18.146003 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" event={"ID":"5fedc871-3ff0-4689-a2e3-29a534cc92dd","Type":"ContainerStarted","Data":"190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e"} Nov 04 02:50:18.282264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:18.280925 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podStartSLOduration=2.280903064 podStartE2EDuration="2.280903064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:18.277567369 +0000 UTC m=+1065.244068363" watchObservedRunningTime="2026-11-04 02:50:18.280903064 +0000 UTC m=+1065.247404061" Nov 04 02:50:19.143901 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:19.143862 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:19.145549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:19.145522 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:19.145660 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:19.145584 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:19.186936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:19.186897 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" event={"ID":"5fedc871-3ff0-4689-a2e3-29a534cc92dd","Type":"ContainerStarted","Data":"067b28692b0619967f07891573b221ccdadc5a4f4dc330d0ecc25c958d0078b8"} Nov 04 02:50:19.299203 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:19.299145 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" podStartSLOduration=2.299130685 podStartE2EDuration="2.299130685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:19.295724242 +0000 UTC m=+1066.262225247" watchObservedRunningTime="2026-11-04 02:50:19.299130685 +0000 UTC m=+1066.265631677" Nov 04 02:50:20.144150 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:20.144117 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:20.144641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:20.144178 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:20.189267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:20.189232 4784 generic.go:356] "Generic (PLEG): container finished" podID="5fedc871-3ff0-4689-a2e3-29a534cc92dd" containerID="067b28692b0619967f07891573b221ccdadc5a4f4dc330d0ecc25c958d0078b8" exitCode=0 Nov 04 02:50:20.189439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:20.189319 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" event={"ID":"5fedc871-3ff0-4689-a2e3-29a534cc92dd","Type":"ContainerDied","Data":"067b28692b0619967f07891573b221ccdadc5a4f4dc330d0ecc25c958d0078b8"} Nov 04 02:50:21.144275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.144234 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:21.144677 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.144291 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:21.394120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.394091 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:21.442407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.442373 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\") pod \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " Nov 04 02:50:21.442600 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.442485 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\" (UniqueName: \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\") pod \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " Nov 04 02:50:21.442674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.442595 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\") pod \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\" (UID: \"5fedc871-3ff0-4689-a2e3-29a534cc92dd\") " Nov 04 02:50:21.442810 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.442787 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca" pod "5fedc871-3ff0-4689-a2e3-29a534cc92dd" (UID: "5fedc871-3ff0-4689-a2e3-29a534cc92dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:50:21.443063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.443047 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:21.444828 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.444792 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca" pod "5fedc871-3ff0-4689-a2e3-29a534cc92dd" (UID: "5fedc871-3ff0-4689-a2e3-29a534cc92dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:50:21.444920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.444872 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv" pod "5fedc871-3ff0-4689-a2e3-29a534cc92dd" (UID: "5fedc871-3ff0-4689-a2e3-29a534cc92dd"). InnerVolumeSpecName "kube-api-access-2k8mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:21.545084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.545035 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8mv\" (UniqueName: \"kubernetes.io/projected/5fedc871-3ff0-4689-a2e3-29a534cc92dd-kube-api-access-2k8mv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:21.545084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:21.545082 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fedc871-3ff0-4689-a2e3-29a534cc92dd-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:22.143847 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:22.143813 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:22.144103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:22.144073 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:22.214580 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:22.214529 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" event={"ID":"5fedc871-3ff0-4689-a2e3-29a534cc92dd","Type":"ContainerDied","Data":"190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e"} Nov 04 02:50:22.215236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:22.215213 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190d501bf3638908d58ca137e4720bd279a559dcb3fc8ecf191c68a1ce67b12e" Nov 04 02:50:22.215439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:22.215426 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t" Nov 04 02:50:23.143813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:23.143781 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:23.144006 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:23.143838 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:24.144055 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:24.144021 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 04 02:50:24.144494 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:24.144080 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 04 02:50:25.148247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:25.148177 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:25.148745 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:25.148723 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:25.149446 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:25.149424 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 02:50:32.486791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.486750 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 02:50:32.487651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.487630 4784 cpu_manager.go:414] "containerMap: removing container" podUID="5fedc871-3ff0-4689-a2e3-29a534cc92dd" containerName="revoke-keys" Nov 04 02:50:32.487746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.487653 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fedc871-3ff0-4689-a2e3-29a534cc92dd" containerName="revoke-keys" Nov 04 02:50:32.487880 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.487866 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fedc871-3ff0-4689-a2e3-29a534cc92dd" containerName="revoke-keys" Nov 04 02:50:32.492686 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.492665 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.602768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.602737 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 02:50:32.617020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.616980 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617152 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617306 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617389 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617441 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617538 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617613 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617578 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617684 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617653 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.617766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.617746 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719289 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719338 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719392 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719448 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719508 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719594 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719657 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719718 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719744 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.719967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.719842 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.720111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.720086 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.720205 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.720135 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.720205 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.720160 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.720341 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.720318 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.721985 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.721961 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.722155 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.722134 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.782851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.782773 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.782851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.782836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:32.806389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:32.806360 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:33.246800 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:33.246772 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de197f_6a69_4a00_91c5_fc25172fc4f4.slice/crio-0558f2dc11d68b2c4d66252f1a1feac948ad25d0b4221e167ba9fc86d6bee2ef WatchSource:0}: Error finding container 0558f2dc11d68b2c4d66252f1a1feac948ad25d0b4221e167ba9fc86d6bee2ef: Status 404 returned error can't find the container with id 0558f2dc11d68b2c4d66252f1a1feac948ad25d0b4221e167ba9fc86d6bee2ef Nov 04 02:50:33.248750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.248724 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 02:50:33.249683 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.249645 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:33.249792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.249765 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:33.249864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.249812 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:33.338515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.338480 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" event={"ID":"29de197f-6a69-4a00-91c5-fc25172fc4f4","Type":"ContainerStarted","Data":"dc74bb1907848c89871e134547ff175d53212ba7bab4f8812d0fb566dda0444e"} Nov 04 02:50:33.338682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.338541 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" event={"ID":"29de197f-6a69-4a00-91c5-fc25172fc4f4","Type":"ContainerStarted","Data":"0558f2dc11d68b2c4d66252f1a1feac948ad25d0b4221e167ba9fc86d6bee2ef"} Nov 04 02:50:33.463340 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.463277 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podStartSLOduration=1.463261875 podStartE2EDuration="1.463261875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:33.461778673 +0000 UTC m=+1080.428279685" watchObservedRunningTime="2026-11-04 02:50:33.463261875 +0000 UTC m=+1080.429762868" Nov 04 02:50:33.806720 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:33.806689 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:34.808931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:34.808897 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 02:50:34.809345 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:34.808951 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 02:50:35.807134 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:35.807095 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 02:50:35.807364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:35.807154 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 02:50:36.417941 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:50:36.417906 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:50:36.807619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:36.807532 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 02:50:36.807619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:36.807586 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 02:50:37.806973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:37.806936 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 02:50:37.807426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:37.806989 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 02:50:38.807526 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:38.807492 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 02:50:38.808021 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:38.807556 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 02:50:39.981644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:39.981440 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:39.982125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:39.982109 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:39.983523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:39.983501 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 02:50:47.348716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.348678 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 02:50:47.353339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.353315 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.449213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449072 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmt82\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449268 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449307 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449356 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449407 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449475 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449573 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449606 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.461025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.449650 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.465745 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.465711 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 02:50:47.551400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551359 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmt82\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551442 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551458 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551487 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551517 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551659 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551787 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551824 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551870 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.551995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.551915 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.552087 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.552009 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.552156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.552131 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.552378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.552359 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.552452 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.552340 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.554028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.554007 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.554355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.554338 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.616845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.616768 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmt82\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.617859 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.617828 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:47.668448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:47.668402 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:48.110643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.110607 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 02:50:48.116053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.116013 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:48.116154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.116097 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:48.116154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.116137 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:50:48.483593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.483551 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" event={"ID":"50fdd56c-88dd-4a2e-9a39-41be0777a0bb","Type":"ContainerStarted","Data":"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f"} Nov 04 02:50:48.483982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.483620 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" event={"ID":"50fdd56c-88dd-4a2e-9a39-41be0777a0bb","Type":"ContainerStarted","Data":"d48c7c28fec3f6071183fbbfc2a995a8ffc4c1605f8dfd8b90a4804f7308ed00"} Nov 04 02:50:48.614970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.614915 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podStartSLOduration=1.614899826 podStartE2EDuration="1.614899826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:48.611478779 +0000 UTC m=+1095.577979772" watchObservedRunningTime="2026-11-04 02:50:48.614899826 +0000 UTC m=+1095.581400817" Nov 04 02:50:48.672359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.672320 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:48.727391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.727353 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:50:48.727604 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:48.727581 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" containerID="cri-o://2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93" gracePeriod=30 Nov 04 02:50:49.677215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:49.677132 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 02:50:49.677750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:49.677267 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 02:50:50.674679 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:50.674636 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 02:50:50.675002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:50.674973 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 02:50:51.676678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:51.676638 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 02:50:51.677327 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:51.677292 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 02:50:52.676737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:52.676691 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 02:50:52.677415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:52.677377 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 02:50:53.680532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:53.677308 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 02:50:53.680532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:53.677379 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 02:50:54.053165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.053140 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:50:54.187418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187383 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187432 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187469 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187513 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187577 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187715 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187742 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.187840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187755 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:50:54.187840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187788 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.188052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187904 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:50:54.188052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187913 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\") pod \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\" (UID: \"fd9d711a-e05a-4165-bf62-ff1ea5deaa34\") " Nov 04 02:50:54.188052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.187949 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:50:54.188409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188157 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:50:54.188497 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188470 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:50:54.188739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188721 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.188812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188745 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.188812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188759 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.188812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188772 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.188812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.188786 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.190139 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.190114 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "kube-api-access-hsq4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:54.190574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.190551 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:50:54.190574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.190559 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:50:54.190730 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.190590 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token" pod "fd9d711a-e05a-4165-bf62-ff1ea5deaa34" (UID: "fd9d711a-e05a-4165-bf62-ff1ea5deaa34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:50:54.290111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.290004 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.290111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.290037 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsq4b\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-kube-api-access-hsq4b\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.290111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.290048 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.290111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.290058 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9d711a-e05a-4165-bf62-ff1ea5deaa34-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:50:54.538767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.538730 4784 generic.go:356] "Generic (PLEG): container finished" podID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerID="2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93" exitCode=0 Nov 04 02:50:54.538973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.538782 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" Nov 04 02:50:54.538973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.538801 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" event={"ID":"fd9d711a-e05a-4165-bf62-ff1ea5deaa34","Type":"ContainerDied","Data":"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93"} Nov 04 02:50:54.538973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.538868 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2" event={"ID":"fd9d711a-e05a-4165-bf62-ff1ea5deaa34","Type":"ContainerDied","Data":"5cf0432ad72dfa1199ae122de3bc10296f104b3ca2fb59fd41e1a1009fa870d1"} Nov 04 02:50:54.538973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.538900 4784 scope.go:122] "RemoveContainer" containerID="2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93" Nov 04 02:50:54.551379 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.551339 4784 scope.go:122] "RemoveContainer" containerID="2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93" Nov 04 02:50:54.551685 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:50:54.551660 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93\": container with ID starting with 2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93 not found: ID does not exist" containerID="2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93" Nov 04 02:50:54.551772 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.551692 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93"} err="failed to get container status \"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93\": rpc error: code = NotFound desc = could not find container \"2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93\": container with ID starting with 2897c8c4ae555525a513c93a916a13807d05370b658aebb6b044f4900c26ac93 not found: ID does not exist" Nov 04 02:50:54.711017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.710981 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:50:54.762369 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.762338 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qx6f2"] Nov 04 02:50:54.873097 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.873017 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:54.873903 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.873871 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:54.874719 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:54.874702 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 02:50:55.164284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:55.164256 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" path="/var/lib/kubelet/pods/fd9d711a-e05a-4165-bf62-ff1ea5deaa34/volumes" Nov 04 02:50:57.603854 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.603817 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 02:50:57.604692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.604660 4784 cpu_manager.go:414] "containerMap: removing container" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" Nov 04 02:50:57.604692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.604690 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" Nov 04 02:50:57.604915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.604903 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9d711a-e05a-4165-bf62-ff1ea5deaa34" containerName="istio-proxy" Nov 04 02:50:57.609909 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.609890 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.665468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.665434 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 02:50:57.711489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.711451 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"kube-root-ca.crt\"" Nov 04 02:50:57.711690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.711548 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"openshift-service-ca.crt\"" Nov 04 02:50:57.711690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.711559 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"distinct-w4-203c35-kserve-self-signed-certs\"" Nov 04 02:50:57.711690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.711561 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"default-dockercfg-tkts4\"" Nov 04 02:50:57.735007 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.734966 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.735261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.735040 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.735261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.735145 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.735261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.735240 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.735437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.735317 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz46j\" (UniqueName: \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.735437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.735394 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837492 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837554 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837604 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837643 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz46j\" (UniqueName: \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837712 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837810 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.837933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837917 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.838035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.837973 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.838142 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.838117 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.840169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.840142 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.840361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.840346 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.900475 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.900399 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz46j\" (UniqueName: \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\") pod \"distinct-w4-203c35-kserve-78f575d697-qclvg\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:57.923633 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:57.923595 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:50:58.365366 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:50:58.365179 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b298ff7_1c44_4582_a93c_c3655e1778cb.slice/crio-f47ccd28211ea81b64d614fa6de530a1a0bb8eb45f17ee134b9a4ddf5e2efecb WatchSource:0}: Error finding container f47ccd28211ea81b64d614fa6de530a1a0bb8eb45f17ee134b9a4ddf5e2efecb: Status 404 returned error can't find the container with id f47ccd28211ea81b64d614fa6de530a1a0bb8eb45f17ee134b9a4ddf5e2efecb Nov 04 02:50:58.365801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:58.365775 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 02:50:58.575343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:58.575293 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" event={"ID":"1b298ff7-1c44-4582-a93c-c3655e1778cb","Type":"ContainerStarted","Data":"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1"} Nov 04 02:50:58.575654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:58.575355 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" event={"ID":"1b298ff7-1c44-4582-a93c-c3655e1778cb","Type":"ContainerStarted","Data":"f47ccd28211ea81b64d614fa6de530a1a0bb8eb45f17ee134b9a4ddf5e2efecb"} Nov 04 02:50:58.700127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:58.700074 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podStartSLOduration=1.70005334 podStartE2EDuration="1.70005334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:58.695578479 +0000 UTC m=+1105.662079475" watchObservedRunningTime="2026-11-04 02:50:58.70005334 +0000 UTC m=+1105.666554349" Nov 04 02:50:58.924244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:50:58.924169 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:51:06.414418 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:51:06.414380 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:51:07.924320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:07.924259 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:51:07.936850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:07.936822 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:51:08.664074 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:08.664034 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:51:12.194617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.194532 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 02:51:12.200026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.200001 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.256969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.256938 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 02:51:12.303337 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.303309 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"distinct-w0-fe4e0c-kserve-self-signed-certs\"" Nov 04 02:51:12.316991 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.316956 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.317172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.317001 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.317172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.317023 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8vz\" (UniqueName: \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.317172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.317119 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.317172 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.317140 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.317357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.317323 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418203 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418311 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418338 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418514 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8vz\" (UniqueName: \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418550 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418584 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418609 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418616 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.418904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.418877 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.421108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.421087 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.421304 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.421289 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.496738 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.496649 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8vz\" (UniqueName: \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\") pod \"distinct-w0-fe4e0c-kserve-d689c4b66-49rpf\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.513974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.513941 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:12.912754 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:51:12.912728 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7fd5de9_0389_4968_876e_4ec6f0ebaf90.slice/crio-17289ad44f0751c12fcfdb0ccafa8758c195ff11a5f5b605c751f968b8e537e0 WatchSource:0}: Error finding container 17289ad44f0751c12fcfdb0ccafa8758c195ff11a5f5b605c751f968b8e537e0: Status 404 returned error can't find the container with id 17289ad44f0751c12fcfdb0ccafa8758c195ff11a5f5b605c751f968b8e537e0 Nov 04 02:51:12.914051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:12.914029 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 02:51:13.696678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:13.696643 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" event={"ID":"e7fd5de9-0389-4968-876e-4ec6f0ebaf90","Type":"ContainerStarted","Data":"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09"} Nov 04 02:51:13.697060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:13.696694 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" event={"ID":"e7fd5de9-0389-4968-876e-4ec6f0ebaf90","Type":"ContainerStarted","Data":"17289ad44f0751c12fcfdb0ccafa8758c195ff11a5f5b605c751f968b8e537e0"} Nov 04 02:51:13.818085 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:13.818018 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podStartSLOduration=1.818001056 podStartE2EDuration="1.818001056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:13.815669554 +0000 UTC m=+1120.782170547" watchObservedRunningTime="2026-11-04 02:51:13.818001056 +0000 UTC m=+1120.784502048" Nov 04 02:51:14.514713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:14.514668 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:15.721015 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:15.720982 4784 generic.go:356] "Generic (PLEG): container finished" podID="d416426e-a063-465e-836c-e6526d784642" containerID="466e517b2c44ea743b14fb48f700c4a9dc1f1bdee8f92785390d980509318d5f" exitCode=0 Nov 04 02:51:15.721537 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:15.721062 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" event={"ID":"d416426e-a063-465e-836c-e6526d784642","Type":"ContainerDied","Data":"466e517b2c44ea743b14fb48f700c4a9dc1f1bdee8f92785390d980509318d5f"} Nov 04 02:51:16.879830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:16.879806 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:51:17.032113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.031993 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\") pod \"d416426e-a063-465e-836c-e6526d784642\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " Nov 04 02:51:17.032377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.032161 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\") pod \"d416426e-a063-465e-836c-e6526d784642\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " Nov 04 02:51:17.032377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.032236 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\") pod \"d416426e-a063-465e-836c-e6526d784642\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " Nov 04 02:51:17.032377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.032324 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\" (UniqueName: \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\") pod \"d416426e-a063-465e-836c-e6526d784642\" (UID: \"d416426e-a063-465e-836c-e6526d784642\") " Nov 04 02:51:17.033227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.033171 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle" pod "d416426e-a063-465e-836c-e6526d784642" (UID: "d416426e-a063-465e-836c-e6526d784642"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:51:17.033815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.033751 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path" pod "d416426e-a063-465e-836c-e6526d784642" (UID: "d416426e-a063-465e-836c-e6526d784642"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:17.033957 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.033835 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle" pod "d416426e-a063-465e-836c-e6526d784642" (UID: "d416426e-a063-465e-836c-e6526d784642"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:51:17.036242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.036212 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx" pod "d416426e-a063-465e-836c-e6526d784642" (UID: "d416426e-a063-465e-836c-e6526d784642"). InnerVolumeSpecName "kube-api-access-xnwgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:51:17.133874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.133835 4784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-trusted-ca-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:17.133874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.133873 4784 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d416426e-a063-465e-836c-e6526d784642-service-ca-bundle\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:17.134089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.133887 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnwgx\" (UniqueName: \"kubernetes.io/projected/d416426e-a063-465e-836c-e6526d784642-kube-api-access-xnwgx\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:17.134089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.133897 4784 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d416426e-a063-465e-836c-e6526d784642-archives-path\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:17.738482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.738442 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" event={"ID":"d416426e-a063-465e-836c-e6526d784642","Type":"ContainerDied","Data":"80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06"} Nov 04 02:51:17.738482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.738481 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44dgj-nwd8w" Nov 04 02:51:17.738482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:17.738491 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80eb08a7405911c05126d4261199334365f39a7d4ed220382d78876b175a6c06" Nov 04 02:51:22.515071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:22.515035 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:22.528373 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:22.528341 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:22.789729 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:22.789646 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:51:36.401574 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:51:36.401536 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:51:40.662653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.662606 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 02:51:40.663891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.663570 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d416426e-a063-465e-836c-e6526d784642" containerName="insights-gathering" Nov 04 02:51:40.663891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.663595 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d416426e-a063-465e-836c-e6526d784642" containerName="insights-gathering" Nov 04 02:51:40.663891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.663755 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d416426e-a063-465e-836c-e6526d784642" containerName="insights-gathering" Nov 04 02:51:40.769791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.766737 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.769791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.769657 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 02:51:40.825804 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.825774 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b7476\"/\"distinct-w5-3b7476-kserve-self-signed-certs\"" Nov 04 02:51:40.841962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.837770 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.841962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.837902 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.841962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.838159 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lgb\" (UniqueName: \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.841962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.838490 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.843234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.842469 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.843234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.842934 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.945840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945791 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.945840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945833 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.946138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945854 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.946138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945897 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.946138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945936 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.946138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.945955 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgb\" (UniqueName: \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.946615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.946595 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.947377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.947330 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.947690 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.947668 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.949891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.949869 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:40.950898 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:40.950877 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:41.009611 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:41.009491 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lgb\" (UniqueName: \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\") pod \"distinct-w5-3b7476-kserve-6df48d8b99-vnxxm\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:41.225272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:41.218495 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:42.430858 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.419904 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 02:51:42.519218 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:51:42.513429 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a23946_92d8_4c17_ac49_799f536621a5.slice/crio-9f702616270d28062db8f0f2439fa8410412125dba8642f53e996245bce049d1 WatchSource:0}: Error finding container 9f702616270d28062db8f0f2439fa8410412125dba8642f53e996245bce049d1: Status 404 returned error can't find the container with id 9f702616270d28062db8f0f2439fa8410412125dba8642f53e996245bce049d1 Nov 04 02:51:42.663342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.663286 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:51:42.714227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.713319 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:51:42.848328 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.848281 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:51:42.890377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.890313 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:51:42.930648 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:42.928328 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:51:43.037403 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:43.037294 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:51:43.359261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:43.359153 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:51:43.398562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:43.396414 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" event={"ID":"15a23946-92d8-4c17-ac49-799f536621a5","Type":"ContainerStarted","Data":"9f702616270d28062db8f0f2439fa8410412125dba8642f53e996245bce049d1"} Nov 04 02:51:43.534210 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:43.533630 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:51:44.405312 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:44.405276 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" event={"ID":"15a23946-92d8-4c17-ac49-799f536621a5","Type":"ContainerStarted","Data":"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3"} Nov 04 02:51:44.526752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:44.526690 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podStartSLOduration=4.526675872 podStartE2EDuration="4.526675872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:44.524502101 +0000 UTC m=+1151.491003093" watchObservedRunningTime="2026-11-04 02:51:44.526675872 +0000 UTC m=+1151.493176864" Nov 04 02:51:45.219619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:45.219585 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:48.489701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.489660 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 02:51:48.494182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.494155 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.576894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.576846 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.576894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.576902 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.576921 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577028 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577057 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6cwk\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577087 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577307 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577434 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577417 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.577494 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.577481 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.605112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.605081 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 02:51:48.678514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678476 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678514 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678518 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678575 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678598 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6cwk\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678642 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678738 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678816 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678868 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.678958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.678944 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.679110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.679060 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.679110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.679091 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.679268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.679150 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.679336 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.679273 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.679486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.679460 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.681421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.681397 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.681769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.681749 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.739691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.739654 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 02:51:48.745025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.744945 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.745212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.745177 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.746449 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.746423 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6cwk\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.780317 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780287 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780335 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780462 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780507 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbmx\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780621 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780665 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780770 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780839 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.780936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.780864 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.816182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.816141 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:48.827899 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.827864 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 02:51:48.881994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.881961 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882005 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882054 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882074 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbmx\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882111 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882148 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882240 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882275 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.882392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882312 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.883011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882630 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.883011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882745 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.883011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882858 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.883011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.882965 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.883415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.883272 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.885269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.885245 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.885359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.885331 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.945794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.945756 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:48.945949 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:48.945881 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbmx\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:49.084532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.084455 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:49.256988 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.256954 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 02:51:49.257840 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:51:49.257814 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4816a13_78c7_4e20_b914_4c589fadfc16.slice/crio-59f9620d38b3a9c4d2c733943cc6522c94ada4aa6e83fdcf5f218a616262162b WatchSource:0}: Error finding container 59f9620d38b3a9c4d2c733943cc6522c94ada4aa6e83fdcf5f218a616262162b: Status 404 returned error can't find the container with id 59f9620d38b3a9c4d2c733943cc6522c94ada4aa6e83fdcf5f218a616262162b Nov 04 02:51:49.260465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.260430 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.260565 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.260502 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.260565 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.260531 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.450202 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.450152 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" event={"ID":"a4816a13-78c7-4e20-b914-4c589fadfc16","Type":"ContainerStarted","Data":"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6"} Nov 04 02:51:49.450415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.450258 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" event={"ID":"a4816a13-78c7-4e20-b914-4c589fadfc16","Type":"ContainerStarted","Data":"59f9620d38b3a9c4d2c733943cc6522c94ada4aa6e83fdcf5f218a616262162b"} Nov 04 02:51:49.468204 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:51:49.468157 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bb2731_4c22_401e_9089_c9f8fe4d50de.slice/crio-a2e519dbb1ef3f129fd0f352d5d38cff04e9ce59e601dbe162df1cf891d18cfe WatchSource:0}: Error finding container a2e519dbb1ef3f129fd0f352d5d38cff04e9ce59e601dbe162df1cf891d18cfe: Status 404 returned error can't find the container with id a2e519dbb1ef3f129fd0f352d5d38cff04e9ce59e601dbe162df1cf891d18cfe Nov 04 02:51:49.471124 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.471096 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 02:51:49.471279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.471252 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.471344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.471319 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.471380 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.471350 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:51:49.576229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.573689 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podStartSLOduration=1.573667002 podStartE2EDuration="1.573667002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:49.572675744 +0000 UTC m=+1156.539176747" watchObservedRunningTime="2026-11-04 02:51:49.573667002 +0000 UTC m=+1156.540167996" Nov 04 02:51:49.816463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:49.816369 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:50.510272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:50.510225 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" event={"ID":"57bb2731-4c22-401e-9089-c9f8fe4d50de","Type":"ContainerStarted","Data":"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377"} Nov 04 02:51:50.510477 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:50.510296 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" event={"ID":"57bb2731-4c22-401e-9089-c9f8fe4d50de","Type":"ContainerStarted","Data":"a2e519dbb1ef3f129fd0f352d5d38cff04e9ce59e601dbe162df1cf891d18cfe"} Nov 04 02:51:50.777878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:50.777751 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podStartSLOduration=2.777729725 podStartE2EDuration="2.777729725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:50.700679401 +0000 UTC m=+1157.667180405" watchObservedRunningTime="2026-11-04 02:51:50.777729725 +0000 UTC m=+1157.744230717" Nov 04 02:51:50.818585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:50.818549 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:50.818781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:50.818630 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:51.092279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.091609 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:51.105243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.099333 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:51.105243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.099398 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:51.221495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.221454 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:51.232238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.232208 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:51.528615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.528582 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:51:51.840003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.834690 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:51.840003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:51.838487 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:52.095278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:52.094837 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:52.095278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:52.094903 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:52.816740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:52.816698 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:52.816935 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:52.816766 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:53.089730 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:53.089628 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:53.090365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:53.090330 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:53.817290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:53.817258 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:53.817478 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:53.817309 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:54.086863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:54.086756 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:54.087151 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:54.087108 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:54.817147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:54.817108 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:54.817682 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:54.817160 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:55.086275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:55.085661 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:55.086275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:55.085728 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:55.817673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:55.817630 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 02:51:55.818078 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:55.817705 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 02:51:56.085261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:56.085148 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 02:51:56.085409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:56.085267 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 02:51:56.821892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:56.821862 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:56.822348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:56.822305 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:56.823043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:56.823023 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 02:51:57.089974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:57.089892 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:57.090238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:57.090217 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:51:57.090918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:51:57.090899 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 02:52:02.939605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:02.939566 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 02:52:02.944521 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:02.944493 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031028 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031165 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031271 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031339 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031401 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031492 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031522 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpnv\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031632 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031538 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.031704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.031687 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.101138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.101104 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 02:52:03.132714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132662 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.132908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132726 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.132908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132828 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.132908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132858 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpnv\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.132908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132889 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.132987 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133011 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133079 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133313 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133147 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133313 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133220 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133472 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.133738 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.133720 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.134381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.134352 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.134530 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.134479 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.136468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.136438 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.137012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.136981 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.196158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.196081 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpnv\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.196526 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.196505 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.259529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.259488 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:03.259529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.259509 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 02:52:03.264020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.263903 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.335973 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjzj\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336032 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336124 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336180 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336241 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336271 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336327 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336423 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.336518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.336461 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.338853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.338823 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 02:52:03.437733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437696 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437738 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437807 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjzj\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437835 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437893 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437938 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.437995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437954 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.437998 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438078 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438139 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438282 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438358 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438510 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.438798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.438768 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.440444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.440425 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.440637 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.440622 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.503346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.503264 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.503480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.503420 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjzj\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.598066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.598028 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:03.708707 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:03.708662 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4f7bc_152f_477b_827f_2099474380bc.slice/crio-4baf1234326b5bf11057ded51bd85d781428e7587bd23a9d7d0908ec220b1cc5 WatchSource:0}: Error finding container 4baf1234326b5bf11057ded51bd85d781428e7587bd23a9d7d0908ec220b1cc5: Status 404 returned error can't find the container with id 4baf1234326b5bf11057ded51bd85d781428e7587bd23a9d7d0908ec220b1cc5 Nov 04 02:52:03.711056 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.710461 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 02:52:03.717825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.717777 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:03.718049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.717878 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:03.718049 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.717930 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:03.989269 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:03.989233 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0589ee_8104_4e43_87f0_d32ab4d8bde1.slice/crio-fefc50ead2f3f0e146ddf2da0dfc4f466c2e43b4fb75ffdc0a06339df293fd16 WatchSource:0}: Error finding container fefc50ead2f3f0e146ddf2da0dfc4f466c2e43b4fb75ffdc0a06339df293fd16: Status 404 returned error can't find the container with id fefc50ead2f3f0e146ddf2da0dfc4f466c2e43b4fb75ffdc0a06339df293fd16 Nov 04 02:52:03.990658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.990632 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 02:52:03.992131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.992095 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:03.992261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.992178 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:03.992261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:03.992247 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:04.631686 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.631647 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" event={"ID":"3f0589ee-8104-4e43-87f0-d32ab4d8bde1","Type":"ContainerStarted","Data":"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68"} Nov 04 02:52:04.631873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.631706 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" event={"ID":"3f0589ee-8104-4e43-87f0-d32ab4d8bde1","Type":"ContainerStarted","Data":"fefc50ead2f3f0e146ddf2da0dfc4f466c2e43b4fb75ffdc0a06339df293fd16"} Nov 04 02:52:04.633654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.633622 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" event={"ID":"f0f4f7bc-152f-477b-827f-2099474380bc","Type":"ContainerStarted","Data":"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56"} Nov 04 02:52:04.633783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.633672 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" event={"ID":"f0f4f7bc-152f-477b-827f-2099474380bc","Type":"ContainerStarted","Data":"4baf1234326b5bf11057ded51bd85d781428e7587bd23a9d7d0908ec220b1cc5"} Nov 04 02:52:04.779787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.779707 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podStartSLOduration=1.779685237 podStartE2EDuration="1.779685237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:04.755239269 +0000 UTC m=+1171.721740262" watchObservedRunningTime="2026-11-04 02:52:04.779685237 +0000 UTC m=+1171.746186227" Nov 04 02:52:04.881212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:04.880812 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podStartSLOduration=2.8807896680000002 podStartE2EDuration="2.880789668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:04.878711956 +0000 UTC m=+1171.845212949" watchObservedRunningTime="2026-11-04 02:52:04.880789668 +0000 UTC m=+1171.847290661" Nov 04 02:52:05.278215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.268559 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:05.290242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.288596 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:05.290242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.288657 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:05.601274 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.600997 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:05.601274 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.601179 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:05.601274 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:05.601243 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:06.264216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:06.263674 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:06.264216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:06.263738 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:06.436920 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:52:06.436886 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:52:06.606273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:06.605584 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:06.606273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:06.605665 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:07.264217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:07.263605 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:07.264217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:07.263681 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:07.604277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:07.601668 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:07.604277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:07.601753 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:08.278215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:08.273416 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:08.278215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:08.273480 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:08.604275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:08.603600 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:08.604275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:08.603672 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:09.268218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:09.264588 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:09.268218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:09.264649 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:09.605662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:09.605567 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:09.606268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:09.606236 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:10.262216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:10.261596 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 02:52:10.262216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:10.261660 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 02:52:10.599272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:10.598913 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 02:52:10.599272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:10.598983 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 02:52:11.376543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.376498 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:11.377500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.377475 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:11.378583 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.378553 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 02:52:11.815982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.815949 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:11.816432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.816397 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:11.817267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:11.817242 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 02:52:15.230419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.230386 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw"] Nov 04 02:52:15.237314 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.237281 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.292939 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.292904 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw"] Nov 04 02:52:15.313525 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.313496 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8vc\" (UniqueName: \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.313692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.313637 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.313744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.313726 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.338141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.338097 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:52:15.338346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.338151 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:52:15.338346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.338154 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:52:15.349052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.349030 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:52:15.415585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.415507 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8vc\" (UniqueName: \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.415784 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.415611 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.415784 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.415709 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.416420 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.416398 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.418336 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.418272 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.479981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.479952 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8vc\" (UniqueName: \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\") pod \"maas-api-revoke-sub-613a3e29-j2btw\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.555334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.555245 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:15.937557 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:15.937515 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99776a70_617e_4514_af4e_feb71ebcef8f.slice/crio-0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05 WatchSource:0}: Error finding container 0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05: Status 404 returned error can't find the container with id 0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05 Nov 04 02:52:15.941000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:15.940938 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw"] Nov 04 02:52:16.794256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:16.794219 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" event={"ID":"99776a70-617e-4514-af4e-feb71ebcef8f","Type":"ContainerStarted","Data":"1507bf897243200d93efcb97691fa1e4e3fb84fa3191dad4c0ab64852f1b1962"} Nov 04 02:52:16.794754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:16.794282 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" event={"ID":"99776a70-617e-4514-af4e-feb71ebcef8f","Type":"ContainerStarted","Data":"0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05"} Nov 04 02:52:16.918886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:16.918829 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" podStartSLOduration=1.918814985 podStartE2EDuration="1.918814985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:16.913681007 +0000 UTC m=+1183.880182025" watchObservedRunningTime="2026-11-04 02:52:16.918814985 +0000 UTC m=+1183.885315976" Nov 04 02:52:17.791352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.791316 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 02:52:17.795767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.795739 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.805253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.805228 4784 generic.go:356] "Generic (PLEG): container finished" podID="99776a70-617e-4514-af4e-feb71ebcef8f" containerID="1507bf897243200d93efcb97691fa1e4e3fb84fa3191dad4c0ab64852f1b1962" exitCode=0 Nov 04 02:52:17.805368 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.805260 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" event={"ID":"99776a70-617e-4514-af4e-feb71ebcef8f","Type":"ContainerDied","Data":"1507bf897243200d93efcb97691fa1e4e3fb84fa3191dad4c0ab64852f1b1962"} Nov 04 02:52:17.851012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.850979 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851016 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851035 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851119 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851317 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851278 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851318 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851340 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851411 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrjp\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.851506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.851495 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.952614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.952581 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.952797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.952634 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.952797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.952753 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.952908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.952798 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrjp\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.952957 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.952912 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953037 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953019 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953103 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953225 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953133 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953225 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953142 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953225 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953167 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953228 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953236 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953452 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.953766 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.953749 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.955549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.955526 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:17.955861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:17.955840 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:18.006533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.006500 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 02:52:18.018017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.017990 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrjp\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:18.018431 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.018405 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:18.111459 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.111366 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:18.544089 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:18.544054 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2bd098_f7bb_4d5c_9c95_1c7dfe73e925.slice/crio-d77bf54d85ae1e374e45cfa57bf67b54cc88deaa811c5acce2e0faf626efab3a WatchSource:0}: Error finding container d77bf54d85ae1e374e45cfa57bf67b54cc88deaa811c5acce2e0faf626efab3a: Status 404 returned error can't find the container with id d77bf54d85ae1e374e45cfa57bf67b54cc88deaa811c5acce2e0faf626efab3a Nov 04 02:52:18.546034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.546013 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 02:52:18.546939 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.546906 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:18.546990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.546975 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:18.547033 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.547007 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:18.816268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.816152 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" event={"ID":"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925","Type":"ContainerStarted","Data":"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596"} Nov 04 02:52:18.816268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.816253 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" event={"ID":"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925","Type":"ContainerStarted","Data":"d77bf54d85ae1e374e45cfa57bf67b54cc88deaa811c5acce2e0faf626efab3a"} Nov 04 02:52:18.947267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:18.947208 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podStartSLOduration=1.9471714530000002 podStartE2EDuration="1.947171453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:18.943709294 +0000 UTC m=+1185.910210287" watchObservedRunningTime="2026-11-04 02:52:18.947171453 +0000 UTC m=+1185.913672447" Nov 04 02:52:19.006642 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.006610 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:19.069535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.069440 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\") pod \"99776a70-617e-4514-af4e-feb71ebcef8f\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " Nov 04 02:52:19.069535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.069518 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\" (UniqueName: \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\") pod \"99776a70-617e-4514-af4e-feb71ebcef8f\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " Nov 04 02:52:19.069783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.069538 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\") pod \"99776a70-617e-4514-af4e-feb71ebcef8f\" (UID: \"99776a70-617e-4514-af4e-feb71ebcef8f\") " Nov 04 02:52:19.069861 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.069837 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca" pod "99776a70-617e-4514-af4e-feb71ebcef8f" (UID: "99776a70-617e-4514-af4e-feb71ebcef8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:52:19.070106 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.070089 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:19.073169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.073131 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca" pod "99776a70-617e-4514-af4e-feb71ebcef8f" (UID: "99776a70-617e-4514-af4e-feb71ebcef8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:52:19.073322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.073283 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc" pod "99776a70-617e-4514-af4e-feb71ebcef8f" (UID: "99776a70-617e-4514-af4e-feb71ebcef8f"). InnerVolumeSpecName "kube-api-access-zh8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:52:19.113498 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.113457 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:19.115459 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.115431 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:19.115616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.115488 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:19.187217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.172690 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8vc\" (UniqueName: \"kubernetes.io/projected/99776a70-617e-4514-af4e-feb71ebcef8f-kube-api-access-zh8vc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:19.187217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.172738 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99776a70-617e-4514-af4e-feb71ebcef8f-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:19.825665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.825630 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" Nov 04 02:52:19.826324 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.825678 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw" event={"ID":"99776a70-617e-4514-af4e-feb71ebcef8f","Type":"ContainerDied","Data":"0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05"} Nov 04 02:52:19.826324 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:19.825725 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c898c987750d7a8f35f788a8afaab42426faa66a0746acb691317324a964c05" Nov 04 02:52:20.112665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:20.112574 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:20.112665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:20.112631 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:21.112014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:21.111980 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:21.112448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:21.112036 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:22.111874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:22.111837 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:22.112089 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:22.111898 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:23.111981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:23.111942 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:23.112171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:23.111996 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:24.112511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:24.112472 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:24.112911 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:24.112531 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:25.111815 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:25.111776 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 02:52:25.112043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:25.111832 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 02:52:26.116706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:26.116674 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:26.117247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:26.117228 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:26.117947 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:26.117924 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 02:52:30.101242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.101205 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27"] Nov 04 02:52:30.101889 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.101875 4784 cpu_manager.go:414] "containerMap: removing container" podUID="99776a70-617e-4514-af4e-feb71ebcef8f" containerName="revoke-keys" Nov 04 02:52:30.101932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.101890 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99776a70-617e-4514-af4e-feb71ebcef8f" containerName="revoke-keys" Nov 04 02:52:30.102007 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.101999 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="99776a70-617e-4514-af4e-feb71ebcef8f" containerName="revoke-keys" Nov 04 02:52:30.106886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.106868 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.159123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.159087 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2vd\" (UniqueName: \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.159637 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.159611 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.159867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.159841 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.163378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.163354 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27"] Nov 04 02:52:30.208091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.208059 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:52:30.208091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.208059 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:52:30.208091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.208070 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:52:30.211650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.211632 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:52:30.260826 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.260795 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.261004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.260859 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.261004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.260909 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2vd\" (UniqueName: \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.261604 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.261582 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.263360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.263328 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.328979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.328945 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2vd\" (UniqueName: \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\") pod \"maas-api-revoke-sub-fd9c3cb5-zqc27\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.421397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.421370 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:30.862147 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:30.862118 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade1dec9_f5ed_479e_a37b_67aaeea9d16c.slice/crio-c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c WatchSource:0}: Error finding container c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c: Status 404 returned error can't find the container with id c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c Nov 04 02:52:30.862993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:30.862966 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27"] Nov 04 02:52:31.029184 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:31.029147 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" event={"ID":"ade1dec9-f5ed-479e-a37b-67aaeea9d16c","Type":"ContainerStarted","Data":"c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c"} Nov 04 02:52:32.043249 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:32.043211 4784 generic.go:356] "Generic (PLEG): container finished" podID="ade1dec9-f5ed-479e-a37b-67aaeea9d16c" containerID="241910685bce8f5d5b5c7baa72211124c2a1f4b659c8975dac39cb1905d88bc5" exitCode=0 Nov 04 02:52:32.043693 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:32.043297 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" event={"ID":"ade1dec9-f5ed-479e-a37b-67aaeea9d16c","Type":"ContainerDied","Data":"241910685bce8f5d5b5c7baa72211124c2a1f4b659c8975dac39cb1905d88bc5"} Nov 04 02:52:33.217781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.217757 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:33.302651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.302569 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\") pod \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " Nov 04 02:52:33.302651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.302613 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\" (UniqueName: \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\") pod \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " Nov 04 02:52:33.302835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.302680 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\") pod \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\" (UID: \"ade1dec9-f5ed-479e-a37b-67aaeea9d16c\") " Nov 04 02:52:33.303124 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.303091 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca" pod "ade1dec9-f5ed-479e-a37b-67aaeea9d16c" (UID: "ade1dec9-f5ed-479e-a37b-67aaeea9d16c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:52:33.303390 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.303372 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:33.304845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.304814 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca" pod "ade1dec9-f5ed-479e-a37b-67aaeea9d16c" (UID: "ade1dec9-f5ed-479e-a37b-67aaeea9d16c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:52:33.304972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.304952 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd" pod "ade1dec9-f5ed-479e-a37b-67aaeea9d16c" (UID: "ade1dec9-f5ed-479e-a37b-67aaeea9d16c"). InnerVolumeSpecName "kube-api-access-kt2vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:52:33.404923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.404890 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:33.404923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.404918 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2vd\" (UniqueName: \"kubernetes.io/projected/ade1dec9-f5ed-479e-a37b-67aaeea9d16c-kube-api-access-kt2vd\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:33.456776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.456749 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:52:33.458582 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.458560 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:52:33.461986 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.461967 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:52:33.470363 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:33.470336 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:52:34.059666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.059628 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" event={"ID":"ade1dec9-f5ed-479e-a37b-67aaeea9d16c","Type":"ContainerDied","Data":"c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c"} Nov 04 02:52:34.059666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.059653 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27" Nov 04 02:52:34.059897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.059705 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c7888e227f2d0fa1d3bdf53fbb66363e58d1a0dfbe1633b297dceada53aa6c" Nov 04 02:52:34.408138 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.408055 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 02:52:34.408819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.408801 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ade1dec9-f5ed-479e-a37b-67aaeea9d16c" containerName="revoke-keys" Nov 04 02:52:34.408905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.408820 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade1dec9-f5ed-479e-a37b-67aaeea9d16c" containerName="revoke-keys" Nov 04 02:52:34.408999 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.408988 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade1dec9-f5ed-479e-a37b-67aaeea9d16c" containerName="revoke-keys" Nov 04 02:52:34.412913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.412896 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.480267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.480228 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 02:52:34.519246 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519178 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519452 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519331 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519478 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519538 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519526 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519543 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519717 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519693 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519802 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngssc\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519834 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.519857 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.519872 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621449 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621512 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621528 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621551 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621634 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngssc\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621652 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621717 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621748 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.621970 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.621825 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.622123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.622080 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.622166 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.622142 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.622320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.622296 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.622373 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.622312 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.622464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.622413 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.624482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.624464 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.624734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.624710 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.689244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.689201 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.689431 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.689411 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngssc\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:34.728101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:34.728060 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:35.162932 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:35.162902 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e72fee_7ec0_4976_83d5_4b7438a0b8c7.slice/crio-c492a9b10ec987d8d7dc80e333c2e80daa1601859f93f7b0e9d8c492d4fdd6ee WatchSource:0}: Error finding container c492a9b10ec987d8d7dc80e333c2e80daa1601859f93f7b0e9d8c492d4fdd6ee: Status 404 returned error can't find the container with id c492a9b10ec987d8d7dc80e333c2e80daa1601859f93f7b0e9d8c492d4fdd6ee Nov 04 02:52:35.165746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:35.165712 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:35.165852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:35.165781 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:35.165852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:35.165810 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:52:35.170304 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:35.170281 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 02:52:36.103239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.100103 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" event={"ID":"22e72fee-7ec0-4976-83d5-4b7438a0b8c7","Type":"ContainerStarted","Data":"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b"} Nov 04 02:52:36.103239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.100208 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" event={"ID":"22e72fee-7ec0-4976-83d5-4b7438a0b8c7","Type":"ContainerStarted","Data":"c492a9b10ec987d8d7dc80e333c2e80daa1601859f93f7b0e9d8c492d4fdd6ee"} Nov 04 02:52:36.285079 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.285018 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podStartSLOduration=2.285003193 podStartE2EDuration="2.285003193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:36.280709148 +0000 UTC m=+1203.247210141" watchObservedRunningTime="2026-11-04 02:52:36.285003193 +0000 UTC m=+1203.251504185" Nov 04 02:52:36.412899 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:52:36.412816 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:52:36.728974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.728933 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:36.730537 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.730511 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:36.730656 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:36.730560 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:37.728432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:37.728395 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:37.728843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:37.728448 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:38.729206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:38.729160 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:38.729609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:38.729281 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:39.728393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:39.728352 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:39.728638 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:39.728617 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:40.728820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:40.728783 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:40.729282 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:40.728843 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:41.728827 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:41.728794 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 02:52:41.729427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:41.728860 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 02:52:42.742900 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:42.742867 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:42.746064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:42.744895 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:42.746291 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:42.746115 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 02:52:51.170399 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.170366 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47"] Nov 04 02:52:51.174873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.174855 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.232801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.232762 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47"] Nov 04 02:52:51.276969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.276932 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:52:51.276969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.276932 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:52:51.277245 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.277001 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:52:51.277669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.277645 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.277756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.277699 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.277983 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.277962 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnphq\" (UniqueName: \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.279943 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.279924 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:52:51.379516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.379481 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.379705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.379529 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.379705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.379633 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnphq\" (UniqueName: \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.380115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.380095 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.382237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.382208 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.442149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.442074 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnphq\" (UniqueName: \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\") pod \"maas-api-revoke-sub-88cd5873-47n47\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.490075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.490043 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:51.876882 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:52:51.876842 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa568d9_644f_4c10_b3af_502c3effc8d2.slice/crio-d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd WatchSource:0}: Error finding container d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd: Status 404 returned error can't find the container with id d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd Nov 04 02:52:51.882564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:51.880782 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47"] Nov 04 02:52:52.320956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:52.320913 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" event={"ID":"0aa568d9-644f-4c10-b3af-502c3effc8d2","Type":"ContainerStarted","Data":"70b4b289be877e4fffa22523bbb372a0b93e0f70fcba22d2f3c35e9de8d0447d"} Nov 04 02:52:52.321398 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:52.320982 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" event={"ID":"0aa568d9-644f-4c10-b3af-502c3effc8d2","Type":"ContainerStarted","Data":"d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd"} Nov 04 02:52:52.443756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:52.443644 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" podStartSLOduration=1.443629635 podStartE2EDuration="1.443629635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:52.438248681 +0000 UTC m=+1219.404749674" watchObservedRunningTime="2026-11-04 02:52:52.443629635 +0000 UTC m=+1219.410130626" Nov 04 02:52:53.331003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:53.330967 4784 generic.go:356] "Generic (PLEG): container finished" podID="0aa568d9-644f-4c10-b3af-502c3effc8d2" containerID="70b4b289be877e4fffa22523bbb372a0b93e0f70fcba22d2f3c35e9de8d0447d" exitCode=0 Nov 04 02:52:53.331491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:53.331053 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" event={"ID":"0aa568d9-644f-4c10-b3af-502c3effc8d2","Type":"ContainerDied","Data":"70b4b289be877e4fffa22523bbb372a0b93e0f70fcba22d2f3c35e9de8d0447d"} Nov 04 02:52:54.496500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.496476 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:54.626314 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.626228 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\" (UniqueName: \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\") pod \"0aa568d9-644f-4c10-b3af-502c3effc8d2\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " Nov 04 02:52:54.626484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.626326 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\") pod \"0aa568d9-644f-4c10-b3af-502c3effc8d2\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " Nov 04 02:52:54.626484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.626368 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\") pod \"0aa568d9-644f-4c10-b3af-502c3effc8d2\" (UID: \"0aa568d9-644f-4c10-b3af-502c3effc8d2\") " Nov 04 02:52:54.626771 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.626726 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca" pod "0aa568d9-644f-4c10-b3af-502c3effc8d2" (UID: "0aa568d9-644f-4c10-b3af-502c3effc8d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:52:54.627348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.627318 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:54.629342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.629308 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca" pod "0aa568d9-644f-4c10-b3af-502c3effc8d2" (UID: "0aa568d9-644f-4c10-b3af-502c3effc8d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:52:54.629436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.629383 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq" pod "0aa568d9-644f-4c10-b3af-502c3effc8d2" (UID: "0aa568d9-644f-4c10-b3af-502c3effc8d2"). InnerVolumeSpecName "kube-api-access-dnphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:52:54.728149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.728114 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnphq\" (UniqueName: \"kubernetes.io/projected/0aa568d9-644f-4c10-b3af-502c3effc8d2-kube-api-access-dnphq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:54.728149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:54.728148 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa568d9-644f-4c10-b3af-502c3effc8d2-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:52:55.349288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:55.349255 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" event={"ID":"0aa568d9-644f-4c10-b3af-502c3effc8d2","Type":"ContainerDied","Data":"d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd"} Nov 04 02:52:55.349466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:55.349307 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d066cf3ad951646eb45b95b7571d3623fdf78768479c0808f9829cc3a2f5cbfd" Nov 04 02:52:55.349466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:55.349271 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47" Nov 04 02:52:59.066518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:59.066480 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8"] Nov 04 02:52:59.117622 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:59.117585 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ffac3b-bkwk8"] Nov 04 02:52:59.161886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:52:59.161852 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5db696c-fb00-4bcc-83f9-e3202989fd01" path="/var/lib/kubelet/pods/d5db696c-fb00-4bcc-83f9-e3202989fd01/volumes" Nov 04 02:53:01.918357 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.918323 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v"] Nov 04 02:53:01.918972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.918958 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0aa568d9-644f-4c10-b3af-502c3effc8d2" containerName="revoke-keys" Nov 04 02:53:01.919019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.918973 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa568d9-644f-4c10-b3af-502c3effc8d2" containerName="revoke-keys" Nov 04 02:53:01.919118 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.919109 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa568d9-644f-4c10-b3af-502c3effc8d2" containerName="revoke-keys" Nov 04 02:53:01.923378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.923353 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:01.981791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:01.981752 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v"] Nov 04 02:53:02.025147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025114 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:53:02.025147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025138 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:53:02.025391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025158 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:53:02.025503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025483 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.025705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025692 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.025741 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.025713 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksq2\" (UniqueName: \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.027855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.027836 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:53:02.127256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.127214 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.127256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.127259 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksq2\" (UniqueName: \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.127485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.127356 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.127905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.127881 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.130019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.129988 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.190374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.190300 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksq2\" (UniqueName: \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\") pod \"maas-api-revoke-sub-4cc6e80c-2b98v\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.237414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.237383 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:02.620853 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:02.620822 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5db053_85b8_468a_b622_c1866ed0d8dd.slice/crio-56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f WatchSource:0}: Error finding container 56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f: Status 404 returned error can't find the container with id 56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f Nov 04 02:53:02.623055 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:02.623034 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v"] Nov 04 02:53:03.415764 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:03.415726 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" event={"ID":"1b5db053-85b8-468a-b622-c1866ed0d8dd","Type":"ContainerStarted","Data":"cb146fc302a8d7348a349f3e546d5a6c7577bd2ae17598f03f327ed136ed9654"} Nov 04 02:53:03.416167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:03.415787 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" event={"ID":"1b5db053-85b8-468a-b622-c1866ed0d8dd","Type":"ContainerStarted","Data":"56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f"} Nov 04 02:53:03.540928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:03.540866 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" podStartSLOduration=2.540846666 podStartE2EDuration="2.540846666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:03.533742267 +0000 UTC m=+1230.500243264" watchObservedRunningTime="2026-11-04 02:53:03.540846666 +0000 UTC m=+1230.507347659" Nov 04 02:53:04.426320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:04.426283 4784 generic.go:356] "Generic (PLEG): container finished" podID="1b5db053-85b8-468a-b622-c1866ed0d8dd" containerID="cb146fc302a8d7348a349f3e546d5a6c7577bd2ae17598f03f327ed136ed9654" exitCode=0 Nov 04 02:53:04.426802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:04.426358 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" event={"ID":"1b5db053-85b8-468a-b622-c1866ed0d8dd","Type":"ContainerDied","Data":"cb146fc302a8d7348a349f3e546d5a6c7577bd2ae17598f03f327ed136ed9654"} Nov 04 02:53:05.605155 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.605130 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:05.680842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.680808 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\") pod \"1b5db053-85b8-468a-b622-c1866ed0d8dd\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " Nov 04 02:53:05.681067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.680988 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\" (UniqueName: \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\") pod \"1b5db053-85b8-468a-b622-c1866ed0d8dd\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " Nov 04 02:53:05.681184 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.681170 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\") pod \"1b5db053-85b8-468a-b622-c1866ed0d8dd\" (UID: \"1b5db053-85b8-468a-b622-c1866ed0d8dd\") " Nov 04 02:53:05.681534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.681511 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca" pod "1b5db053-85b8-468a-b622-c1866ed0d8dd" (UID: "1b5db053-85b8-468a-b622-c1866ed0d8dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:53:05.681994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.681972 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:05.683461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.683432 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca" pod "1b5db053-85b8-468a-b622-c1866ed0d8dd" (UID: "1b5db053-85b8-468a-b622-c1866ed0d8dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:53:05.683658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.683636 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2" pod "1b5db053-85b8-468a-b622-c1866ed0d8dd" (UID: "1b5db053-85b8-468a-b622-c1866ed0d8dd"). InnerVolumeSpecName "kube-api-access-cksq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:53:05.783670 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.783624 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5db053-85b8-468a-b622-c1866ed0d8dd-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:05.783670 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:05.783655 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksq2\" (UniqueName: \"kubernetes.io/projected/1b5db053-85b8-468a-b622-c1866ed0d8dd-kube-api-access-cksq2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:06.401935 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:53:06.401897 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:53:06.447116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.447076 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" Nov 04 02:53:06.447354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.447118 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v" event={"ID":"1b5db053-85b8-468a-b622-c1866ed0d8dd","Type":"ContainerDied","Data":"56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f"} Nov 04 02:53:06.447354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.447176 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e27d729b9913321913962c071dcf2a9aa828dbdcb08370665a3a40eefd358f" Nov 04 02:53:06.904214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.889446 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 02:53:06.904214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.895840 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1b5db053-85b8-468a-b622-c1866ed0d8dd" containerName="revoke-keys" Nov 04 02:53:06.904214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.895864 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5db053-85b8-468a-b622-c1866ed0d8dd" containerName="revoke-keys" Nov 04 02:53:06.904214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.896089 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5db053-85b8-468a-b622-c1866ed0d8dd" containerName="revoke-keys" Nov 04 02:53:06.911213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.908712 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:06.949938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.948293 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 02:53:06.968206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:06.962613 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"distinct-2-w4-203c35-kserve-self-signed-certs\"" Nov 04 02:53:07.005630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.005585 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.005836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.005662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.005836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.005690 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.005836 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.005818 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.005997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.005918 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.006103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.006084 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z9t\" (UniqueName: \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.108706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.108636 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.108953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.108831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z9t\" (UniqueName: \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.108953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.108880 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.108953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.108937 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.109120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.108967 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.109120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.109101 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.110305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.109682 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.110305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.109963 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.114704 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.114683 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.114874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.114866 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.118841 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.118734 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.175326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.175286 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z9t\" (UniqueName: \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\") pod \"distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.242732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.242684 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:07.729262 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:07.729218 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod464cc908_f59f_4ce3_9dd1_96a11ebf3188.slice/crio-8e0357c3c6396d1d190a9eb38570c68bd039c383b74a28e04d15ed9f7fe0775b WatchSource:0}: Error finding container 8e0357c3c6396d1d190a9eb38570c68bd039c383b74a28e04d15ed9f7fe0775b: Status 404 returned error can't find the container with id 8e0357c3c6396d1d190a9eb38570c68bd039c383b74a28e04d15ed9f7fe0775b Nov 04 02:53:07.744826 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:07.744790 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 02:53:08.530001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:08.529945 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" event={"ID":"464cc908-f59f-4ce3-9dd1-96a11ebf3188","Type":"ContainerStarted","Data":"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442"} Nov 04 02:53:08.530001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:08.530002 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" event={"ID":"464cc908-f59f-4ce3-9dd1-96a11ebf3188","Type":"ContainerStarted","Data":"8e0357c3c6396d1d190a9eb38570c68bd039c383b74a28e04d15ed9f7fe0775b"} Nov 04 02:53:08.651150 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:08.651091 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podStartSLOduration=2.651075322 podStartE2EDuration="2.651075322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:08.649439786 +0000 UTC m=+1235.615940780" watchObservedRunningTime="2026-11-04 02:53:08.651075322 +0000 UTC m=+1235.617576313" Nov 04 02:53:09.242989 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:09.242953 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:14.051046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:14.051003 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr"] Nov 04 02:53:14.101895 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:14.101844 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d9216-n6gsr"] Nov 04 02:53:15.161959 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:15.161922 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207741ba-6201-43b0-a615-873051f60d97" path="/var/lib/kubelet/pods/207741ba-6201-43b0-a615-873051f60d97/volumes" Nov 04 02:53:17.243561 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:17.243521 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:17.255824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:17.255797 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:17.626872 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:17.626776 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:20.105590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.105549 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 02:53:20.110260 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.110240 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.209238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.209150 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 02:53:20.224004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.223969 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224126 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224350 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224477 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224395 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224477 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224444 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qln4\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224558 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224793 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.224867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.224835 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.321739 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.321699 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss"] Nov 04 02:53:20.326058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326025 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326078 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326103 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qln4\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326140 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326249 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326303 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326342 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326425 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326512 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326557 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.326937 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.326913 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.327091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.327067 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.327170 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.327036 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.327249 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.327215 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.327746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.327718 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.329992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.329970 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.330301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.330277 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.382314 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.382221 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss"] Nov 04 02:53:20.383736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.383698 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:53:20.384052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.383979 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:53:20.384052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.384031 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:53:20.384052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.384031 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:53:20.391060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.391036 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.391163 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.391093 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qln4\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.425620 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.425587 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:20.428405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.428363 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.428602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.428571 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.428713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.428666 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pwg\" (UniqueName: \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.530577 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.530540 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pwg\" (UniqueName: \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.530751 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.530730 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.530918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.530900 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.532232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.531799 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.533893 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.533851 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.593464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.593426 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pwg\" (UniqueName: \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\") pod \"maas-api-revoke-sub-5928b909-hlbss\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.658297 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.658175 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:20.863246 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:20.863216 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb7646f_3fac_49f1_b138_a77250cc41d4.slice/crio-1b32058262e2d89676ad8c4236e798919eb1b2ff480ec99bdee816419cb0f434 WatchSource:0}: Error finding container 1b32058262e2d89676ad8c4236e798919eb1b2ff480ec99bdee816419cb0f434: Status 404 returned error can't find the container with id 1b32058262e2d89676ad8c4236e798919eb1b2ff480ec99bdee816419cb0f434 Nov 04 02:53:20.865627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.865599 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 02:53:20.866141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.866110 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:20.866227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.866173 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:20.866264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:20.866225 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:21.042098 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:21.042070 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41124fed_a849_4f3f_827b_9fbd3ac92d70.slice/crio-ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420 WatchSource:0}: Error finding container ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420: Status 404 returned error can't find the container with id ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420 Nov 04 02:53:21.043120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.043090 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss"] Nov 04 02:53:21.654128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.654029 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" event={"ID":"41124fed-a849-4f3f-827b-9fbd3ac92d70","Type":"ContainerStarted","Data":"923edf75f0b5417e60becd63146df039c2faa5e20de4f20101ffe800eb664cd5"} Nov 04 02:53:21.654128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.654089 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" event={"ID":"41124fed-a849-4f3f-827b-9fbd3ac92d70","Type":"ContainerStarted","Data":"ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420"} Nov 04 02:53:21.656361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.656331 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" event={"ID":"6bb7646f-3fac-49f1-b138-a77250cc41d4","Type":"ContainerStarted","Data":"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42"} Nov 04 02:53:21.656543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.656387 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" event={"ID":"6bb7646f-3fac-49f1-b138-a77250cc41d4","Type":"ContainerStarted","Data":"1b32058262e2d89676ad8c4236e798919eb1b2ff480ec99bdee816419cb0f434"} Nov 04 02:53:21.782926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.782851 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" podStartSLOduration=1.7828265 podStartE2EDuration="1.7828265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:21.77485432 +0000 UTC m=+1248.741355313" watchObservedRunningTime="2026-11-04 02:53:21.7828265 +0000 UTC m=+1248.749327494" Nov 04 02:53:21.898546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:21.898489 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podStartSLOduration=1.8984721119999999 podStartE2EDuration="1.898472112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:21.89654429 +0000 UTC m=+1248.863045294" watchObservedRunningTime="2026-11-04 02:53:21.898472112 +0000 UTC m=+1248.864973105" Nov 04 02:53:22.426467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:22.426429 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:22.427953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:22.427919 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:22.428083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:22.427965 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:22.690597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:22.690558 4784 generic.go:356] "Generic (PLEG): container finished" podID="41124fed-a849-4f3f-827b-9fbd3ac92d70" containerID="923edf75f0b5417e60becd63146df039c2faa5e20de4f20101ffe800eb664cd5" exitCode=0 Nov 04 02:53:22.691103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:22.690637 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" event={"ID":"41124fed-a849-4f3f-827b-9fbd3ac92d70","Type":"ContainerDied","Data":"923edf75f0b5417e60becd63146df039c2faa5e20de4f20101ffe800eb664cd5"} Nov 04 02:53:23.426347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:23.426313 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:23.426803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:23.426368 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:23.940917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:23.940887 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:24.121616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.121527 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\" (UniqueName: \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\") pod \"41124fed-a849-4f3f-827b-9fbd3ac92d70\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " Nov 04 02:53:24.121800 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.121606 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\") pod \"41124fed-a849-4f3f-827b-9fbd3ac92d70\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " Nov 04 02:53:24.121901 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.121864 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\") pod \"41124fed-a849-4f3f-827b-9fbd3ac92d70\" (UID: \"41124fed-a849-4f3f-827b-9fbd3ac92d70\") " Nov 04 02:53:24.122032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.121973 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca" pod "41124fed-a849-4f3f-827b-9fbd3ac92d70" (UID: "41124fed-a849-4f3f-827b-9fbd3ac92d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:53:24.122491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.122465 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:24.123925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.123896 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg" pod "41124fed-a849-4f3f-827b-9fbd3ac92d70" (UID: "41124fed-a849-4f3f-827b-9fbd3ac92d70"). InnerVolumeSpecName "kube-api-access-72pwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:53:24.124042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.123977 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca" pod "41124fed-a849-4f3f-827b-9fbd3ac92d70" (UID: "41124fed-a849-4f3f-827b-9fbd3ac92d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:53:24.224099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.224060 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41124fed-a849-4f3f-827b-9fbd3ac92d70-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:24.224099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.224096 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pwg\" (UniqueName: \"kubernetes.io/projected/41124fed-a849-4f3f-827b-9fbd3ac92d70-kube-api-access-72pwg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:24.426768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.426728 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:24.426956 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.426794 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:24.741437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.741344 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" event={"ID":"41124fed-a849-4f3f-827b-9fbd3ac92d70","Type":"ContainerDied","Data":"ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420"} Nov 04 02:53:24.741437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.741402 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8c63577981d656e08cc1fea1da2545b5091fbcf5c64d747e8d63bc98eb6420" Nov 04 02:53:24.741689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:24.741479 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss" Nov 04 02:53:25.427303 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:25.426784 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:25.427303 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:25.426847 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:26.430436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.429839 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:26.430436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.429914 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:26.521216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.520294 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:53:26.624484 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.624441 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:53:26.668594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.668544 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:53:26.761182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.761078 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:26.793214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.791395 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:26.793214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.792429 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:53:26.844554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:26.843762 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.010528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.010480 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.366271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.366222 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.428651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.428618 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:27.428935 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.428899 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:27.524417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.524354 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.525518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.525496 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.801548 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.801918 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.802486 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.802541 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.806178 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.807725 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.810409 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.810762 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.811720 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:53:27.815430 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.815021 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:53:27.872770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.871956 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.872770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.872391 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:53:27.915221 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.912546 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:27.915221 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.912612 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.929578 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:27.929549 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:53:28.128093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.127239 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 02:53:28.194750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.194685 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:53:28.209547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.209522 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:53:28.245084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.245036 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.263129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.261594 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:28.263461 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.263422 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.535236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.479364 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.535236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.481288 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:53:28.535236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.506724 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:53:28.570277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.569115 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.606296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.589911 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:28.606296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.589969 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:28.613169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.611601 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:53:28.821270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.815335 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:53:28.821270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.816831 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.846753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.846710 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:53:28.852314 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.852273 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.978117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.932775 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.978570 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:28.943707 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.134635 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.134552 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:53:29.134877 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.134863 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:53:29.134992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.134982 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:53:29.135107 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.135093 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 02:53:29.137415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.137391 4784 cpu_manager.go:414] "containerMap: removing container" podUID="41124fed-a849-4f3f-827b-9fbd3ac92d70" containerName="revoke-keys" Nov 04 02:53:29.139438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.139416 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41124fed-a849-4f3f-827b-9fbd3ac92d70" containerName="revoke-keys" Nov 04 02:53:29.139851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.139833 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="41124fed-a849-4f3f-827b-9fbd3ac92d70" containerName="revoke-keys" Nov 04 02:53:29.145245 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.145230 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.214235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.209270 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"default-dockercfg-dp8tf\"" Nov 04 02:53:29.214235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.209357 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"openshift-service-ca.crt\"" Nov 04 02:53:29.214235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.209462 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"distinct-2-w3-cdcf67-kserve-self-signed-certs\"" Nov 04 02:53:29.214235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.209519 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"kube-root-ca.crt\"" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.305970 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.306081 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.306144 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.306289 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jsg\" (UniqueName: \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.306344 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.307270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.306374 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.347359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.336984 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 02:53:29.347359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.337070 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:53:29.347359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.337103 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:53:29.347359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.337135 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:53:29.408334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408247 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408289 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408723 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408755 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408792 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408825 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.408920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.408901 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jsg\" (UniqueName: \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.409955 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.409923 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.410083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.410017 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.412237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.412176 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.418600 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.416619 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.432652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.432610 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 02:53:29.432828 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.432684 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 02:53:29.472133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.472102 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jsg\" (UniqueName: \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\") pod \"distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:29.657709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:29.657668 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:30.083492 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.083439 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6"] Nov 04 02:53:30.097508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.086095 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 02:53:30.097508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.094543 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d06e4-fnrp6"] Nov 04 02:53:30.918034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.917998 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:30.918545 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.918444 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:30.919563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:30.919540 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 02:53:31.014681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:31.014644 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" event={"ID":"28572dfe-e4cc-4127-b9b5-ad50dbd85367","Type":"ContainerStarted","Data":"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad"} Nov 04 02:53:31.014853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:31.014692 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" event={"ID":"28572dfe-e4cc-4127-b9b5-ad50dbd85367","Type":"ContainerStarted","Data":"3125ae7df3db0e41da56d5827452d76040b4e0cc65d77a8d7fc451879e7405fd"} Nov 04 02:53:31.161974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:31.161935 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68de16b5-94b2-4587-8b70-60f42fdab656" path="/var/lib/kubelet/pods/68de16b5-94b2-4587-8b70-60f42fdab656/volumes" Nov 04 02:53:31.163219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:31.163155 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podStartSLOduration=3.163144093 podStartE2EDuration="3.163144093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:31.162915124 +0000 UTC m=+1258.129416119" watchObservedRunningTime="2026-11-04 02:53:31.163144093 +0000 UTC m=+1258.129645084" Nov 04 02:53:31.658543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:31.658512 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:36.072346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.072303 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz"] Nov 04 02:53:36.078574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.078549 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.134215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.134159 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz"] Nov 04 02:53:36.146105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.146071 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w86b\" (UniqueName: \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.146306 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.146262 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.146491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.146466 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.178623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.178586 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:53:36.178808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.178642 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:53:36.178808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.178647 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:53:36.182925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.182907 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:53:36.248290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.248247 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.248493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.248418 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w86b\" (UniqueName: \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.248710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.248680 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.249072 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.249046 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.251115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.251093 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.311863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.311828 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w86b\" (UniqueName: \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\") pod \"maas-api-revoke-sub-0d602a9d-g7mgz\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.395867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.395789 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:36.403244 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:53:36.403209 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:53:36.780445 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:36.780418 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33db7759_85cd_4888_a49b_59a2b5aef2ee.slice/crio-dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890 WatchSource:0}: Error finding container dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890: Status 404 returned error can't find the container with id dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890 Nov 04 02:53:36.782384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:36.782357 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz"] Nov 04 02:53:37.086231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:37.086099 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" event={"ID":"33db7759-85cd-4888-a49b-59a2b5aef2ee","Type":"ContainerStarted","Data":"dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890"} Nov 04 02:53:38.097042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:38.096957 4784 generic.go:356] "Generic (PLEG): container finished" podID="33db7759-85cd-4888-a49b-59a2b5aef2ee" containerID="1d2c65b77630804af34dbf1adc716b7a41eb30379f8e44d854ab9f46ddbcc652" exitCode=0 Nov 04 02:53:38.097042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:38.097005 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" event={"ID":"33db7759-85cd-4888-a49b-59a2b5aef2ee","Type":"ContainerDied","Data":"1d2c65b77630804af34dbf1adc716b7a41eb30379f8e44d854ab9f46ddbcc652"} Nov 04 02:53:39.283375 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.283346 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:39.398485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.398398 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\" (UniqueName: \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\") pod \"33db7759-85cd-4888-a49b-59a2b5aef2ee\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " Nov 04 02:53:39.398668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.398653 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\") pod \"33db7759-85cd-4888-a49b-59a2b5aef2ee\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " Nov 04 02:53:39.398740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.398674 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\") pod \"33db7759-85cd-4888-a49b-59a2b5aef2ee\" (UID: \"33db7759-85cd-4888-a49b-59a2b5aef2ee\") " Nov 04 02:53:39.399145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.399108 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca" pod "33db7759-85cd-4888-a49b-59a2b5aef2ee" (UID: "33db7759-85cd-4888-a49b-59a2b5aef2ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:53:39.401744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.401714 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b" pod "33db7759-85cd-4888-a49b-59a2b5aef2ee" (UID: "33db7759-85cd-4888-a49b-59a2b5aef2ee"). InnerVolumeSpecName "kube-api-access-5w86b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:53:39.402288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.402260 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca" pod "33db7759-85cd-4888-a49b-59a2b5aef2ee" (UID: "33db7759-85cd-4888-a49b-59a2b5aef2ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:53:39.500162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.500125 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:39.500162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.500156 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33db7759-85cd-4888-a49b-59a2b5aef2ee-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:39.500162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.500168 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w86b\" (UniqueName: \"kubernetes.io/projected/33db7759-85cd-4888-a49b-59a2b5aef2ee-kube-api-access-5w86b\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:53:39.658716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.658610 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:39.672439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:39.672405 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:40.116855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:40.116823 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" Nov 04 02:53:40.116855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:40.116849 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz" event={"ID":"33db7759-85cd-4888-a49b-59a2b5aef2ee","Type":"ContainerDied","Data":"dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890"} Nov 04 02:53:40.117077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:40.116921 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd241676b86a540ca82c9a35b85c4f673d4a32aed5977abc0d9ccc15080de890" Nov 04 02:53:40.129256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:40.129230 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:53:49.415050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.415016 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 02:53:49.415750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.415736 4784 cpu_manager.go:414] "containerMap: removing container" podUID="33db7759-85cd-4888-a49b-59a2b5aef2ee" containerName="revoke-keys" Nov 04 02:53:49.415812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.415752 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33db7759-85cd-4888-a49b-59a2b5aef2ee" containerName="revoke-keys" Nov 04 02:53:49.415866 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.415857 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="33db7759-85cd-4888-a49b-59a2b5aef2ee" containerName="revoke-keys" Nov 04 02:53:49.419481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.419454 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450579 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450632 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450708 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450814 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450857 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.450915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.450890 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.451071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.451038 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.451124 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.451075 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.451124 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.451104 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxb2\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.530436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.530400 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 02:53:49.552057 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552022 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552266 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552247 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552281 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552328 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552519 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552599 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552569 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552602 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552610 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxb2\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552883 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.552910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552901 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.553050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.552975 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.553424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.553396 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.553531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.553432 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.554077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.554043 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.555730 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.555696 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.555819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.555751 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.616989 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.616960 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.617201 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.617166 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxb2\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:49.734520 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:49.734426 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:50.179839 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:53:50.179811 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef92cdc_24ad_48c1_a4eb_23fe719f4719.slice/crio-8588d90c0ec59a17218307e5d76dd5590756b921bde7916f71623d51a36099f4 WatchSource:0}: Error finding container 8588d90c0ec59a17218307e5d76dd5590756b921bde7916f71623d51a36099f4: Status 404 returned error can't find the container with id 8588d90c0ec59a17218307e5d76dd5590756b921bde7916f71623d51a36099f4 Nov 04 02:53:50.180912 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:50.180890 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 02:53:50.182390 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:50.182359 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:50.182440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:50.182422 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:50.182471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:50.182450 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:53:50.207619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:50.207590 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" event={"ID":"3ef92cdc-24ad-48c1-a4eb-23fe719f4719","Type":"ContainerStarted","Data":"8588d90c0ec59a17218307e5d76dd5590756b921bde7916f71623d51a36099f4"} Nov 04 02:53:51.053298 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.053260 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h"] Nov 04 02:53:51.103777 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.103744 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de56d712-2ll7h"] Nov 04 02:53:51.164875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.164836 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9779eee6-26df-49b3-9350-180c17ef2675" path="/var/lib/kubelet/pods/9779eee6-26df-49b3-9350-180c17ef2675/volumes" Nov 04 02:53:51.218344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.218309 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" event={"ID":"3ef92cdc-24ad-48c1-a4eb-23fe719f4719","Type":"ContainerStarted","Data":"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051"} Nov 04 02:53:51.354036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.353917 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podStartSLOduration=2.353903303 podStartE2EDuration="2.353903303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:51.348617896 +0000 UTC m=+1278.315118889" watchObservedRunningTime="2026-11-04 02:53:51.353903303 +0000 UTC m=+1278.320404295" Nov 04 02:53:51.735486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.735452 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:51.737243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.737223 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:51.737309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:51.737279 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:52.735831 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:52.735792 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:52.736387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:52.735862 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:53.734843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:53.734812 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:53.735063 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:53.734865 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:54.735476 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:54.735436 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:54.735906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:54.735501 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:55.735578 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:55.735541 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:55.736002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:55.735609 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:56.735174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:56.735137 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 02:53:56.735409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:56.735238 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 02:53:58.178813 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:58.178780 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:58.179701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:58.179678 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:53:58.180812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:53:58.180789 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 02:54:02.070962 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.070905 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr"] Nov 04 02:54:02.273728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.273685 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36fb0a0-h5hsr"] Nov 04 02:54:02.313169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.312307 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:54:02.622923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.622855 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:54:02.832217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.825326 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.832217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.826262 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:54:02.855355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.855293 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.934392 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.934342 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.984358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:02.984307 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.029342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.029293 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.091816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.091773 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.131585 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.131294 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.298931 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.298839 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f50898-36e2-43a4-bcd7-2c0053587d60" path="/var/lib/kubelet/pods/d7f50898-36e2-43a4-bcd7-2c0053587d60/volumes" Nov 04 02:54:03.334828 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.334782 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:54:03.339216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.335748 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:54:03.515215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.511135 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.529343 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.529294 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.646399 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.646290 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.650213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.647011 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:54:03.833495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.833447 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:54:03.835386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.835359 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:54:03.855914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.842371 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:54:03.858612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.858586 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:54:03.911769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.859467 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.911769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.859741 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:54:03.940581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.940540 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.959698 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.993301 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:03.993660 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.051549 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.051918 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.092636 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.097212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.093896 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:54:04.237453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.223435 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:54:04.237453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.223831 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:54:04.401793 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.397309 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.752224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.740299 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.876229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.872728 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.917274 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.877738 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:54:04.941613 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.886335 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.983341 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:04.983292 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.006864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.006766 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:54:05.066218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.056753 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:54:05.095663 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.095619 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.284275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.283851 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:54:05.465305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.465264 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:54:05.749420 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.749375 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.878440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.878394 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.966250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.947289 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.966250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:05.948130 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.050308 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.045954 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.050308 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.047621 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.094215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.090320 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.114305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.096681 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.233249 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:54:06.222470 4784 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.008s" Nov 04 02:54:06.247417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.247376 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 02:54:06.611879 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.611836 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:54:07.086050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.870858 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.090093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.894474 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.093850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:06.999121 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.134643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.134596 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:54:07.146390 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.146345 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.273471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.273441 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.283714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.273040 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:54:07.313160 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:54:07.313122 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:54:07.342008 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.341923 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.349755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.349715 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:07.351751 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.351704 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:54:07.382327 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.382299 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 02:54:07.384518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.384496 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:54:07.384727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.384715 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:54:07.384855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.384843 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:54:07.384987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.384975 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:54:07.385103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.385094 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:54:07.385251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.385238 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:54:07.386305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.386284 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:54:07.386551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.386531 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:54:07.471294 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471257 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvq7\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471487 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471460 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471508 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471637 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471616 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471696 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471659 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471878 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471852 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471921 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.471988 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.471951 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.472161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.472141 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574384 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574453 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574653 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574702 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574812 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574903 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574878 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.574958 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.574902 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.575111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.575092 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.575467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.575448 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvq7\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.575851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.575830 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.575940 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.575914 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.577110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.576476 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.577110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.576555 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.577372 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.577344 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.578046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.578020 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.578272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.578252 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.637819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.637736 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.637819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.637780 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvq7\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.693471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.693436 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:07.967645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.967599 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl"] Nov 04 02:54:07.973652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.973622 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:07.983742 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.983709 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zt7j\" (UniqueName: \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:07.984323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.984003 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:07.984323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:07.984038 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.028269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.028230 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl"] Nov 04 02:54:08.029162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.029135 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:54:08.029500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.029471 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:54:08.029615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.029577 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:54:08.029752 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.029724 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:54:08.085545 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.085510 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.085735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.085553 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.085735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.085695 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zt7j\" (UniqueName: \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.090479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.088394 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.094427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.094395 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.165088 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:08.165052 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15eff77_b23b_42aa_82b4_a69e34eee6cb.slice/crio-6b4e707b5162556c45bc79825d256b527a71f7c2fa784b9af2f3c6250742b9f6 WatchSource:0}: Error finding container 6b4e707b5162556c45bc79825d256b527a71f7c2fa784b9af2f3c6250742b9f6: Status 404 returned error can't find the container with id 6b4e707b5162556c45bc79825d256b527a71f7c2fa784b9af2f3c6250742b9f6 Nov 04 02:54:08.167967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.167940 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zt7j\" (UniqueName: \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\") pod \"maas-api-revoke-sub-3e6d834a-cfnvl\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.168080 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.168050 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 02:54:08.168355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.168323 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:08.168448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.168391 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:08.168448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.168419 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:08.296429 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.296388 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:08.438053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.428245 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" event={"ID":"c15eff77-b23b-42aa-82b4-a69e34eee6cb","Type":"ContainerStarted","Data":"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c"} Nov 04 02:54:08.438053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.428400 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" event={"ID":"c15eff77-b23b-42aa-82b4-a69e34eee6cb","Type":"ContainerStarted","Data":"6b4e707b5162556c45bc79825d256b527a71f7c2fa784b9af2f3c6250742b9f6"} Nov 04 02:54:08.694180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.694144 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:08.783816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.776492 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl"] Nov 04 02:54:08.791002 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:08.790963 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75b5dee_2c99_418a_a4c7_e5a52637559f.slice/crio-a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55 WatchSource:0}: Error finding container a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55: Status 404 returned error can't find the container with id a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55 Nov 04 02:54:08.879171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:08.879115 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podStartSLOduration=4.879099018 podStartE2EDuration="4.879099018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:08.873727909 +0000 UTC m=+1295.840228903" watchObservedRunningTime="2026-11-04 02:54:08.879099018 +0000 UTC m=+1295.845600010" Nov 04 02:54:09.253102 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.253020 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv"] Nov 04 02:54:09.260628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.260590 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.313994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.313960 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv"] Nov 04 02:54:09.314792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.314762 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8f64\" (UniqueName: \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.314946 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.314886 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.315014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.314953 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.416856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.416816 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8f64\" (UniqueName: \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.417068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.416924 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.417068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.416997 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.418535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.418502 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.428654 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.428621 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.452672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.449601 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" event={"ID":"c75b5dee-2c99-418a-a4c7-e5a52637559f","Type":"ContainerStarted","Data":"be1904b2107de301c9e5d427fb02f54100dfb0898e6e388ba2b38cfcb28917a0"} Nov 04 02:54:09.452672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.449699 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" event={"ID":"c75b5dee-2c99-418a-a4c7-e5a52637559f","Type":"ContainerStarted","Data":"a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55"} Nov 04 02:54:09.489803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.489770 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8f64\" (UniqueName: \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\") pod \"maas-api-revoke-sub-9522d074-q4mwv\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.591017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.590936 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:09.697218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.696684 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:09.697218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.696737 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:09.987746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.987371 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" podStartSLOduration=2.987356288 podStartE2EDuration="2.987356288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:09.582530953 +0000 UTC m=+1296.549031947" watchObservedRunningTime="2026-11-04 02:54:09.987356288 +0000 UTC m=+1296.953857280" Nov 04 02:54:09.990355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:09.990332 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv"] Nov 04 02:54:10.010430 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:10.010392 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2a451b_6b8d_404a_9439_a629987a7b59.slice/crio-ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26 WatchSource:0}: Error finding container ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26: Status 404 returned error can't find the container with id ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26 Nov 04 02:54:10.086386 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.085266 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6"] Nov 04 02:54:10.095902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.095862 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119bc78b-sqtl6"] Nov 04 02:54:10.485658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.485627 4784 generic.go:356] "Generic (PLEG): container finished" podID="c75b5dee-2c99-418a-a4c7-e5a52637559f" containerID="be1904b2107de301c9e5d427fb02f54100dfb0898e6e388ba2b38cfcb28917a0" exitCode=0 Nov 04 02:54:10.485868 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.485712 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" event={"ID":"c75b5dee-2c99-418a-a4c7-e5a52637559f","Type":"ContainerDied","Data":"be1904b2107de301c9e5d427fb02f54100dfb0898e6e388ba2b38cfcb28917a0"} Nov 04 02:54:10.487238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.487201 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" event={"ID":"1d2a451b-6b8d-404a-9439-a629987a7b59","Type":"ContainerStarted","Data":"ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26"} Nov 04 02:54:10.694950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.694918 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:10.695136 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:10.694977 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:11.188691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.188649 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa15464-02c6-4231-b818-3ff6c1ba5c4c" path="/var/lib/kubelet/pods/9fa15464-02c6-4231-b818-3ff6c1ba5c4c/volumes" Nov 04 02:54:11.498451 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.498364 4784 generic.go:356] "Generic (PLEG): container finished" podID="1d2a451b-6b8d-404a-9439-a629987a7b59" containerID="469d8131cf3d00f54265dbbf205e22d319c1039b5c0ff9ac28521e3f9fa3be15" exitCode=0 Nov 04 02:54:11.498607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.498446 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" event={"ID":"1d2a451b-6b8d-404a-9439-a629987a7b59","Type":"ContainerDied","Data":"469d8131cf3d00f54265dbbf205e22d319c1039b5c0ff9ac28521e3f9fa3be15"} Nov 04 02:54:11.694131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.694108 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:11.694321 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.694156 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:11.726643 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.726620 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:11.751361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.751277 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\") pod \"c75b5dee-2c99-418a-a4c7-e5a52637559f\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " Nov 04 02:54:11.751535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.751520 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\" (UniqueName: \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\") pod \"c75b5dee-2c99-418a-a4c7-e5a52637559f\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " Nov 04 02:54:11.751624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.751610 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\") pod \"c75b5dee-2c99-418a-a4c7-e5a52637559f\" (UID: \"c75b5dee-2c99-418a-a4c7-e5a52637559f\") " Nov 04 02:54:11.752054 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.752021 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca" pod "c75b5dee-2c99-418a-a4c7-e5a52637559f" (UID: "c75b5dee-2c99-418a-a4c7-e5a52637559f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:11.752480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.752360 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:11.753649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.753624 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca" pod "c75b5dee-2c99-418a-a4c7-e5a52637559f" (UID: "c75b5dee-2c99-418a-a4c7-e5a52637559f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:11.753852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.753835 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j" pod "c75b5dee-2c99-418a-a4c7-e5a52637559f" (UID: "c75b5dee-2c99-418a-a4c7-e5a52637559f"). InnerVolumeSpecName "kube-api-access-9zt7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:11.853645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.853610 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75b5dee-2c99-418a-a4c7-e5a52637559f-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:11.853645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:11.853643 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zt7j\" (UniqueName: \"kubernetes.io/projected/c75b5dee-2c99-418a-a4c7-e5a52637559f-kube-api-access-9zt7j\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:12.509298 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.509264 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" Nov 04 02:54:12.509754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.509314 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl" event={"ID":"c75b5dee-2c99-418a-a4c7-e5a52637559f","Type":"ContainerDied","Data":"a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55"} Nov 04 02:54:12.509754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.509376 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b53c1a9b6ca9ead88845257790ea81870a9aa29279de4d78f5dd6a7c832d55" Nov 04 02:54:12.693843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.693810 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:12.694043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.693874 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:12.728859 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.728833 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:12.791016 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.790915 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\") pod \"1d2a451b-6b8d-404a-9439-a629987a7b59\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " Nov 04 02:54:12.791016 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.790957 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\") pod \"1d2a451b-6b8d-404a-9439-a629987a7b59\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " Nov 04 02:54:12.791356 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.791241 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\" (UniqueName: \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\") pod \"1d2a451b-6b8d-404a-9439-a629987a7b59\" (UID: \"1d2a451b-6b8d-404a-9439-a629987a7b59\") " Nov 04 02:54:12.791446 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.791423 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca" pod "1d2a451b-6b8d-404a-9439-a629987a7b59" (UID: "1d2a451b-6b8d-404a-9439-a629987a7b59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:12.792042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.792020 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:12.794294 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.794258 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca" pod "1d2a451b-6b8d-404a-9439-a629987a7b59" (UID: "1d2a451b-6b8d-404a-9439-a629987a7b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:12.794455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.794429 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64" pod "1d2a451b-6b8d-404a-9439-a629987a7b59" (UID: "1d2a451b-6b8d-404a-9439-a629987a7b59"). InnerVolumeSpecName "kube-api-access-j8f64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:12.893832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.893728 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2a451b-6b8d-404a-9439-a629987a7b59-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:12.893832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:12.893761 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8f64\" (UniqueName: \"kubernetes.io/projected/1d2a451b-6b8d-404a-9439-a629987a7b59-kube-api-access-j8f64\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:13.519558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:13.519479 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" event={"ID":"1d2a451b-6b8d-404a-9439-a629987a7b59","Type":"ContainerDied","Data":"ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26"} Nov 04 02:54:13.519558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:13.519521 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv" Nov 04 02:54:13.520002 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:13.519526 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1e8ed4f36f5fe7df8102ee91d6216666346955cbebd595ce42264094fc2f26" Nov 04 02:54:13.694770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:13.694737 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:13.694967 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:13.694797 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:14.693879 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:14.693844 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 02:54:14.694283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:14.693917 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 02:54:16.169504 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:16.169448 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:16.170044 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:16.170006 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:16.172014 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:16.171990 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 02:54:20.104658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.104625 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69"] Nov 04 02:54:20.105665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105648 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1d2a451b-6b8d-404a-9439-a629987a7b59" containerName="revoke-keys" Nov 04 02:54:20.105714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105668 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2a451b-6b8d-404a-9439-a629987a7b59" containerName="revoke-keys" Nov 04 02:54:20.105746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105715 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c75b5dee-2c99-418a-a4c7-e5a52637559f" containerName="revoke-keys" Nov 04 02:54:20.105746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105723 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75b5dee-2c99-418a-a4c7-e5a52637559f" containerName="revoke-keys" Nov 04 02:54:20.105893 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105882 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75b5dee-2c99-418a-a4c7-e5a52637559f" containerName="revoke-keys" Nov 04 02:54:20.105928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.105904 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2a451b-6b8d-404a-9439-a629987a7b59" containerName="revoke-keys" Nov 04 02:54:20.109746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.109725 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.165339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.165307 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69"] Nov 04 02:54:20.203616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.203586 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.203777 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.203736 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpclj\" (UniqueName: \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.203838 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.203812 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.208550 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.208516 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:54:20.208550 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.208545 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:54:20.208729 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.208589 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:54:20.214502 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.214475 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:54:20.305569 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.305535 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.305728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.305642 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpclj\" (UniqueName: \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.305728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.305711 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.306463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.306442 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.308374 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.308348 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.368479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.368413 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpclj\" (UniqueName: \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\") pod \"maas-api-revoke-sub-64713a1b-gdr69\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.424678 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.424645 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:20.811947 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:20.811915 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5169994d_87fc_4bbe_acd7_328d809eefdb.slice/crio-7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1 WatchSource:0}: Error finding container 7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1: Status 404 returned error can't find the container with id 7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1 Nov 04 02:54:20.813159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:20.813135 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69"] Nov 04 02:54:21.601496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:21.601442 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" event={"ID":"5169994d-87fc-4bbe-acd7-328d809eefdb","Type":"ContainerStarted","Data":"cf10de6cc13e8aa4edc0485045982ccd7b1704c9f20e07e5d4ef677283a45ae1"} Nov 04 02:54:21.601960 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:21.601511 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" event={"ID":"5169994d-87fc-4bbe-acd7-328d809eefdb","Type":"ContainerStarted","Data":"7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1"} Nov 04 02:54:22.612132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:22.612097 4784 generic.go:356] "Generic (PLEG): container finished" podID="5169994d-87fc-4bbe-acd7-328d809eefdb" containerID="cf10de6cc13e8aa4edc0485045982ccd7b1704c9f20e07e5d4ef677283a45ae1" exitCode=0 Nov 04 02:54:22.612592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:22.612177 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" event={"ID":"5169994d-87fc-4bbe-acd7-328d809eefdb","Type":"ContainerDied","Data":"cf10de6cc13e8aa4edc0485045982ccd7b1704c9f20e07e5d4ef677283a45ae1"} Nov 04 02:54:23.805825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.805794 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:23.965300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.965260 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\" (UniqueName: \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\") pod \"5169994d-87fc-4bbe-acd7-328d809eefdb\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " Nov 04 02:54:23.965610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.965538 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\") pod \"5169994d-87fc-4bbe-acd7-328d809eefdb\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " Nov 04 02:54:23.965610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.965576 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\") pod \"5169994d-87fc-4bbe-acd7-328d809eefdb\" (UID: \"5169994d-87fc-4bbe-acd7-328d809eefdb\") " Nov 04 02:54:23.965951 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.965922 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca" pod "5169994d-87fc-4bbe-acd7-328d809eefdb" (UID: "5169994d-87fc-4bbe-acd7-328d809eefdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:23.966378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.966357 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:23.968256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.968228 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj" pod "5169994d-87fc-4bbe-acd7-328d809eefdb" (UID: "5169994d-87fc-4bbe-acd7-328d809eefdb"). InnerVolumeSpecName "kube-api-access-qpclj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:23.968402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.968381 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca" pod "5169994d-87fc-4bbe-acd7-328d809eefdb" (UID: "5169994d-87fc-4bbe-acd7-328d809eefdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:23.999066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:23.999023 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr"] Nov 04 02:54:24.000028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.000001 4784 cpu_manager.go:414] "containerMap: removing container" podUID="5169994d-87fc-4bbe-acd7-328d809eefdb" containerName="revoke-keys" Nov 04 02:54:24.000028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.000029 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5169994d-87fc-4bbe-acd7-328d809eefdb" containerName="revoke-keys" Nov 04 02:54:24.000265 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.000251 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5169994d-87fc-4bbe-acd7-328d809eefdb" containerName="revoke-keys" Nov 04 02:54:24.004624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.004605 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.062749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.062710 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr"] Nov 04 02:54:24.068241 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.068207 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5169994d-87fc-4bbe-acd7-328d809eefdb-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:24.068415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.068244 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpclj\" (UniqueName: \"kubernetes.io/projected/5169994d-87fc-4bbe-acd7-328d809eefdb-kube-api-access-qpclj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:24.169711 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.169671 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.169985 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.169961 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.170060 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.170020 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pv6\" (UniqueName: \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.271716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.271623 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.271716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.271681 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pv6\" (UniqueName: \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.271950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.271843 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.272346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.272324 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.274839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.274813 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.334391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.334362 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pv6\" (UniqueName: \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\") pod \"maas-api-revoke-sub-78a4bbe8-vtdqr\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.620107 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.620028 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:24.630990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.630956 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" event={"ID":"5169994d-87fc-4bbe-acd7-328d809eefdb","Type":"ContainerDied","Data":"7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1"} Nov 04 02:54:24.631127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.631017 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acada0fa1095696b6909f259443f32b8473dd2a7554af5900b58f8d8e046ae1" Nov 04 02:54:24.631127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:24.630976 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69" Nov 04 02:54:25.005315 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:25.005273 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74eb776f_a59f_4f98_ac03_cb396a461c3f.slice/crio-34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78 WatchSource:0}: Error finding container 34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78: Status 404 returned error can't find the container with id 34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78 Nov 04 02:54:25.006344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:25.006311 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr"] Nov 04 02:54:25.642061 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:25.641963 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" event={"ID":"74eb776f-a59f-4f98-ac03-cb396a461c3f","Type":"ContainerStarted","Data":"5017c1497023922f0ae81d150d5dea927b0d9e9f005b12619f7efe73677086fd"} Nov 04 02:54:25.642061 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:25.642036 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" event={"ID":"74eb776f-a59f-4f98-ac03-cb396a461c3f","Type":"ContainerStarted","Data":"34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78"} Nov 04 02:54:25.762988 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:25.762933 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" podStartSLOduration=2.762919789 podStartE2EDuration="2.762919789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:25.759992358 +0000 UTC m=+1312.726493350" watchObservedRunningTime="2026-11-04 02:54:25.762919789 +0000 UTC m=+1312.729420781" Nov 04 02:54:26.652787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:26.652748 4784 generic.go:356] "Generic (PLEG): container finished" podID="74eb776f-a59f-4f98-ac03-cb396a461c3f" containerID="5017c1497023922f0ae81d150d5dea927b0d9e9f005b12619f7efe73677086fd" exitCode=0 Nov 04 02:54:26.653260 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:26.652833 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" event={"ID":"74eb776f-a59f-4f98-ac03-cb396a461c3f","Type":"ContainerDied","Data":"5017c1497023922f0ae81d150d5dea927b0d9e9f005b12619f7efe73677086fd"} Nov 04 02:54:27.085744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.085705 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9"] Nov 04 02:54:27.090441 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.090413 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.150022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.149984 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9"] Nov 04 02:54:27.217641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.217606 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9xh\" (UniqueName: \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.217790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.217651 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.217790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.217725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.319946 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.319907 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9xh\" (UniqueName: \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.320157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.319952 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.320157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.319996 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.321034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.321008 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.322824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.322801 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.382927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.382852 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9xh\" (UniqueName: \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\") pod \"maas-api-revoke-sub-050eadcc-9hpp9\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.407012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.406974 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:27.799282 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:27.799243 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1275091_41fd_45eb_9468_90a6fded50a8.slice/crio-b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a WatchSource:0}: Error finding container b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a: Status 404 returned error can't find the container with id b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a Nov 04 02:54:27.800479 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.800454 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9"] Nov 04 02:54:27.825625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.825599 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:27.826915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.826892 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\") pod \"74eb776f-a59f-4f98-ac03-cb396a461c3f\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " Nov 04 02:54:27.826978 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.826955 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\" (UniqueName: \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\") pod \"74eb776f-a59f-4f98-ac03-cb396a461c3f\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " Nov 04 02:54:27.827302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.827173 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\") pod \"74eb776f-a59f-4f98-ac03-cb396a461c3f\" (UID: \"74eb776f-a59f-4f98-ac03-cb396a461c3f\") " Nov 04 02:54:27.827420 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.827308 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca" pod "74eb776f-a59f-4f98-ac03-cb396a461c3f" (UID: "74eb776f-a59f-4f98-ac03-cb396a461c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:27.827854 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.827762 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:27.829300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.829272 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6" pod "74eb776f-a59f-4f98-ac03-cb396a461c3f" (UID: "74eb776f-a59f-4f98-ac03-cb396a461c3f"). InnerVolumeSpecName "kube-api-access-26pv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:27.829379 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.829275 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca" pod "74eb776f-a59f-4f98-ac03-cb396a461c3f" (UID: "74eb776f-a59f-4f98-ac03-cb396a461c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:27.929669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.929634 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pv6\" (UniqueName: \"kubernetes.io/projected/74eb776f-a59f-4f98-ac03-cb396a461c3f-kube-api-access-26pv6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:27.929669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:27.929666 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74eb776f-a59f-4f98-ac03-cb396a461c3f-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:28.054864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.054831 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2"] Nov 04 02:54:28.062886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.062849 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b13839-d6kz2"] Nov 04 02:54:28.672177 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.672137 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" event={"ID":"c1275091-41fd-45eb-9468-90a6fded50a8","Type":"ContainerStarted","Data":"41267eb7636a790047f7f5143c60c8ea9da09af01a2b788746de804c4862ee1a"} Nov 04 02:54:28.672381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.672220 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" event={"ID":"c1275091-41fd-45eb-9468-90a6fded50a8","Type":"ContainerStarted","Data":"b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a"} Nov 04 02:54:28.673557 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.673536 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" Nov 04 02:54:28.673681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.673585 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr" event={"ID":"74eb776f-a59f-4f98-ac03-cb396a461c3f","Type":"ContainerDied","Data":"34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78"} Nov 04 02:54:28.673681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.673624 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f00d5aa2e9dd42fd84868eace77d91f9f5948fe32c3b92cc534e47e8df0a78" Nov 04 02:54:28.791612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:28.791565 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" podStartSLOduration=1.7915511670000002 podStartE2EDuration="1.791551167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:28.791111703 +0000 UTC m=+1315.757612696" watchObservedRunningTime="2026-11-04 02:54:28.791551167 +0000 UTC m=+1315.758052158" Nov 04 02:54:29.051854 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.051768 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm"] Nov 04 02:54:29.056909 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.056882 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc59bfb-cglsm"] Nov 04 02:54:29.162084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.162052 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e579e29-8f23-432a-9ea2-6232f656c1f6" path="/var/lib/kubelet/pods/2e579e29-8f23-432a-9ea2-6232f656c1f6/volumes" Nov 04 02:54:29.162649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.162631 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f793cc-05d3-428b-a4a2-d36eb9950f42" path="/var/lib/kubelet/pods/87f793cc-05d3-428b-a4a2-d36eb9950f42/volumes" Nov 04 02:54:29.683873 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.683835 4784 generic.go:356] "Generic (PLEG): container finished" podID="c1275091-41fd-45eb-9468-90a6fded50a8" containerID="41267eb7636a790047f7f5143c60c8ea9da09af01a2b788746de804c4862ee1a" exitCode=0 Nov 04 02:54:29.684052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:29.683915 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" event={"ID":"c1275091-41fd-45eb-9468-90a6fded50a8","Type":"ContainerDied","Data":"41267eb7636a790047f7f5143c60c8ea9da09af01a2b788746de804c4862ee1a"} Nov 04 02:54:30.884041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.884012 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:30.985316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.985177 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\") pod \"c1275091-41fd-45eb-9468-90a6fded50a8\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " Nov 04 02:54:30.985472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.985335 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\" (UniqueName: \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\") pod \"c1275091-41fd-45eb-9468-90a6fded50a8\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " Nov 04 02:54:30.985472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.985432 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\") pod \"c1275091-41fd-45eb-9468-90a6fded50a8\" (UID: \"c1275091-41fd-45eb-9468-90a6fded50a8\") " Nov 04 02:54:30.985771 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.985748 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca" pod "c1275091-41fd-45eb-9468-90a6fded50a8" (UID: "c1275091-41fd-45eb-9468-90a6fded50a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:30.986107 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.986057 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:30.987587 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.987552 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca" pod "c1275091-41fd-45eb-9468-90a6fded50a8" (UID: "c1275091-41fd-45eb-9468-90a6fded50a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:30.987680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:30.987652 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh" pod "c1275091-41fd-45eb-9468-90a6fded50a8" (UID: "c1275091-41fd-45eb-9468-90a6fded50a8"). InnerVolumeSpecName "kube-api-access-jb9xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:31.061559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.061512 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp"] Nov 04 02:54:31.066909 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.066880 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4db5d8-psnlp"] Nov 04 02:54:31.087767 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.087736 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1275091-41fd-45eb-9468-90a6fded50a8-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:31.087932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.087785 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9xh\" (UniqueName: \"kubernetes.io/projected/c1275091-41fd-45eb-9468-90a6fded50a8-kube-api-access-jb9xh\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:31.163548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.163513 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6eb948-dd47-434e-9000-e32177d3903a" path="/var/lib/kubelet/pods/bd6eb948-dd47-434e-9000-e32177d3903a/volumes" Nov 04 02:54:31.705260 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.705233 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" Nov 04 02:54:31.705463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.705221 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9" event={"ID":"c1275091-41fd-45eb-9468-90a6fded50a8","Type":"ContainerDied","Data":"b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a"} Nov 04 02:54:31.705463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:31.705366 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b049f7ff7f22ea64849ac2fa5211da3a7dc7bd6f71c87a9f48bc6974ea638c3a" Nov 04 02:54:33.383900 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:33.383864 4784 scope.go:122] "RemoveContainer" containerID="b8c23407319a051f63aaf2efdad0ce735d3731b54fbfde5d59334bfe04a22708" Nov 04 02:54:33.395938 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:33.395913 4784 scope.go:122] "RemoveContainer" containerID="c0302cbc773840e852b0b8a7ccad6c80fa1823d028dfc1d5f0a74ff28b4aebd1" Nov 04 02:54:33.407985 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:33.407963 4784 scope.go:122] "RemoveContainer" containerID="54f613292b07fdfe4267522e2e2f724fc701be59bf22da10a2bddfdf0060d8ad" Nov 04 02:54:34.053733 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.053698 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67"] Nov 04 02:54:34.058738 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.058711 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df2e539-pkm67"] Nov 04 02:54:34.402867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.402780 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql"] Nov 04 02:54:34.403541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403526 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c1275091-41fd-45eb-9468-90a6fded50a8" containerName="revoke-keys" Nov 04 02:54:34.403593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403543 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1275091-41fd-45eb-9468-90a6fded50a8" containerName="revoke-keys" Nov 04 02:54:34.403593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403555 4784 cpu_manager.go:414] "containerMap: removing container" podUID="74eb776f-a59f-4f98-ac03-cb396a461c3f" containerName="revoke-keys" Nov 04 02:54:34.403593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403560 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74eb776f-a59f-4f98-ac03-cb396a461c3f" containerName="revoke-keys" Nov 04 02:54:34.403689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403670 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1275091-41fd-45eb-9468-90a6fded50a8" containerName="revoke-keys" Nov 04 02:54:34.403689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.403679 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="74eb776f-a59f-4f98-ac03-cb396a461c3f" containerName="revoke-keys" Nov 04 02:54:34.409458 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.409437 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.463051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.463016 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:54:34.463051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.463020 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:54:34.463345 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.463239 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:54:34.513869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.513834 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:54:34.520336 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.520300 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql"] Nov 04 02:54:34.540639 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.540599 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8jxg\" (UniqueName: \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.540810 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.540759 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.541000 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.540979 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.642845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.642805 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.643025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.642966 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8jxg\" (UniqueName: \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.643025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.643013 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.643648 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.643628 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.645732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.645699 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.705354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.705320 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8jxg\" (UniqueName: \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\") pod \"maas-api-revoke-sub-7b97258c-j9lql\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:34.727046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:34.727020 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:35.283817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.281039 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6c3f1d-b725-455a-93df-e86110e4a3f6" path="/var/lib/kubelet/pods/ac6c3f1d-b725-455a-93df-e86110e4a3f6/volumes" Nov 04 02:54:35.283817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.281564 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql"] Nov 04 02:54:35.450539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.450416 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 02:54:35.469028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.469001 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.622053 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628231 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628284 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628321 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628451 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628540 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jvw\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628600 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628625 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628654 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.644817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.628695 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.732942 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733067 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jvw\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733134 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733161 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733209 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733250 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733357 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733404 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.733462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.733436 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.734427 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.734035 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.734694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.734666 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.736043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.735759 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.736043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.735776 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.736487 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.736460 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.755515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.755465 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" event={"ID":"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae","Type":"ContainerStarted","Data":"9c4230d11842d2693bc1ba678799c5f06400d9d0c1f5035bc8d18e0282f5c053"} Nov 04 02:54:35.776982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.776720 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.776982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.776930 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.843679 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.839424 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jvw\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:35.845908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:35.845879 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:36.045867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:36.045481 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:37.002214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.001285 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.067216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.012632 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.112936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.112888 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.193280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.155132 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:54:37.381685 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.381633 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.515422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.398533 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:54:37.516890 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:54:37.496402 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:54:37.838296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.564640 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:54:37.838296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.798389 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.838296 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:37.815909 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dd8e0e_f14c_4b80_93ed_569fc70494e4.slice/crio-473a5299099373ec1b4adb4295fc71c671e9610065b4bd9a1fe186ab659e0d53 WatchSource:0}: Error finding container 473a5299099373ec1b4adb4295fc71c671e9610065b4bd9a1fe186ab659e0d53: Status 404 returned error can't find the container with id 473a5299099373ec1b4adb4295fc71c671e9610065b4bd9a1fe186ab659e0d53 Nov 04 02:54:37.857218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.855572 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:37.857218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.855682 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:37.857218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.855732 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:37.888433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.888245 4784 generic.go:356] "Generic (PLEG): container finished" podID="3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" containerID="feecfe2d2f21cb8e930547d82a4731fcb2ca78ff8b9ff17083a0b6d5b4f91c84" exitCode=0 Nov 04 02:54:37.888511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.888463 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" event={"ID":"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae","Type":"ContainerDied","Data":"feecfe2d2f21cb8e930547d82a4731fcb2ca78ff8b9ff17083a0b6d5b4f91c84"} Nov 04 02:54:37.931745 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:37.923654 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 02:54:38.004467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.002029 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.004467 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.002838 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:54:38.016213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.014684 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.016213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.016132 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:54:38.057456 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.056954 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:54:38.117414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.113630 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.117414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.113994 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 02:54:38.117414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.114058 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:54:38.117414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.115217 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:54:38.206268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.194390 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:54:38.277271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.269374 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.326099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.325257 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.326099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.325592 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:54:38.416812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.409646 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:54:38.506735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.503169 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.512272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.507264 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:54:38.519708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.519441 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.525922 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.523039 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:54:38.585212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.581334 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 04 02:54:38.585212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.581443 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.602936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.601331 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:54:38.799331 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.799181 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.800680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.800653 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:54:38.813281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.813240 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f"] Nov 04 02:54:38.825367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.825341 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:38.935289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:38.934253 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.014200 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.014148 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.022650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.022617 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9kb\" (UniqueName: \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.023504 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.023479 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.023843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.019126 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:54:39.030433 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.030410 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f"] Nov 04 02:54:39.056769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.056679 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" event={"ID":"47dd8e0e-f14c-4b80-93ed-569fc70494e4","Type":"ContainerStarted","Data":"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09"} Nov 04 02:54:39.057024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.057001 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" event={"ID":"47dd8e0e-f14c-4b80-93ed-569fc70494e4","Type":"ContainerStarted","Data":"473a5299099373ec1b4adb4295fc71c671e9610065b4bd9a1fe186ab659e0d53"} Nov 04 02:54:39.060574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.060552 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:39.060847 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.060828 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:39.060995 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.060970 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:39.085218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.085179 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:54:39.118263 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.116750 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.118263 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.117085 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:54:39.120352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.118602 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:54:39.237247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.178143 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9kb\" (UniqueName: \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.237247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.178271 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.237247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.178567 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.237247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.205269 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:54:39.357774 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.354730 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.413351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.412686 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.550533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.550500 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9kb\" (UniqueName: \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\") pod \"maas-api-revoke-sub-fda9d425-ts85f\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.660151 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.651283 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.664064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.663789 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:39.782332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.780806 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:54:39.846734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.845706 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.927351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.924737 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podStartSLOduration=4.924697244 podStartE2EDuration="4.924697244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:39.521755736 +0000 UTC m=+1326.488256723" watchObservedRunningTime="2026-11-04 02:54:39.924697244 +0000 UTC m=+1326.891198237" Nov 04 02:54:39.959036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.958993 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:54:39.959123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.959086 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:54:39.959174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.959141 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:54:39.959247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.959215 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:54:39.959300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.959265 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:54:39.959354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:39.959320 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:54:40.009179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.009082 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:54:40.010268 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.010247 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:54:40.098426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.051826 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:40.098426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.051886 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:40.208248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.207821 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:40.332703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.332666 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\") pod \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " Nov 04 02:54:40.333017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.332999 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\") pod \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " Nov 04 02:54:40.333083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.333049 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\" (UniqueName: \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\") pod \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\" (UID: \"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae\") " Nov 04 02:54:40.333120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.333071 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca" pod "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" (UID: "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:40.333831 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.333743 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:40.335362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.335337 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca" pod "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" (UID: "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:40.335513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.335494 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg" pod "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" (UID: "3caa815b-a11b-4d03-88ae-d2f2a4c1dbae"). InnerVolumeSpecName "kube-api-access-d8jxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:40.391232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.379843 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f"] Nov 04 02:54:40.435031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.435000 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:40.435285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:40.435261 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8jxg\" (UniqueName: \"kubernetes.io/projected/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae-kube-api-access-d8jxg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:41.051131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.051038 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:41.051131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.051092 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:41.132623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.132586 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" event={"ID":"75df7aec-8910-4f5e-b60f-fe0239debfc3","Type":"ContainerStarted","Data":"680f6b167f705c8a3093a4f133625b5af76bdb20eee3d33a553cbcd6294b7b38"} Nov 04 02:54:41.133098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.132695 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" event={"ID":"75df7aec-8910-4f5e-b60f-fe0239debfc3","Type":"ContainerStarted","Data":"19efb8ea61299a62bc775fb0a28b95df5089d3659d22d38a8474af66e22f08d1"} Nov 04 02:54:41.134513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.134468 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" event={"ID":"3caa815b-a11b-4d03-88ae-d2f2a4c1dbae","Type":"ContainerDied","Data":"9c4230d11842d2693bc1ba678799c5f06400d9d0c1f5035bc8d18e0282f5c053"} Nov 04 02:54:41.134513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.134488 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql" Nov 04 02:54:41.134703 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.134529 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c4230d11842d2693bc1ba678799c5f06400d9d0c1f5035bc8d18e0282f5c053" Nov 04 02:54:41.487080 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:41.487020 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" podStartSLOduration=3.487004269 podStartE2EDuration="3.487004269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:41.482482361 +0000 UTC m=+1328.448983355" watchObservedRunningTime="2026-11-04 02:54:41.487004269 +0000 UTC m=+1328.453505310" Nov 04 02:54:42.030094 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.030059 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx"] Nov 04 02:54:42.030935 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.030918 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" containerName="revoke-keys" Nov 04 02:54:42.031022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.030939 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" containerName="revoke-keys" Nov 04 02:54:42.031153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.031138 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" containerName="revoke-keys" Nov 04 02:54:42.035074 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.035053 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.050923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.050895 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:42.051059 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.050942 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:42.082503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.082463 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.082882 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.082859 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.083243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.083220 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75ld\" (UniqueName: \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.091563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.091523 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx"] Nov 04 02:54:42.146023 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.145989 4784 generic.go:356] "Generic (PLEG): container finished" podID="75df7aec-8910-4f5e-b60f-fe0239debfc3" containerID="680f6b167f705c8a3093a4f133625b5af76bdb20eee3d33a553cbcd6294b7b38" exitCode=0 Nov 04 02:54:42.146488 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.146073 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" event={"ID":"75df7aec-8910-4f5e-b60f-fe0239debfc3","Type":"ContainerDied","Data":"680f6b167f705c8a3093a4f133625b5af76bdb20eee3d33a553cbcd6294b7b38"} Nov 04 02:54:42.186083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.186030 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.186350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.186328 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75ld\" (UniqueName: \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.186507 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.186484 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.186908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.186873 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.189869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.189845 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.248776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.248739 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75ld\" (UniqueName: \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\") pod \"maas-api-revoke-sub-117e506e-gxhtx\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.351762 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.351664 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:42.821840 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:42.821806 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b1d22_3665_4ad5_912e_fb3afaf5f314.slice/crio-ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572 WatchSource:0}: Error finding container ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572: Status 404 returned error can't find the container with id ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572 Nov 04 02:54:42.826732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:42.826701 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx"] Nov 04 02:54:43.051371 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.051337 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:43.051558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.051390 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:43.190347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.190318 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" event={"ID":"383b1d22-3665-4ad5-912e-fb3afaf5f314","Type":"ContainerStarted","Data":"ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572"} Nov 04 02:54:43.435933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.435908 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:43.520377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.520339 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\") pod \"75df7aec-8910-4f5e-b60f-fe0239debfc3\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " Nov 04 02:54:43.520556 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.520460 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\") pod \"75df7aec-8910-4f5e-b60f-fe0239debfc3\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " Nov 04 02:54:43.520556 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.520519 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\" (UniqueName: \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\") pod \"75df7aec-8910-4f5e-b60f-fe0239debfc3\" (UID: \"75df7aec-8910-4f5e-b60f-fe0239debfc3\") " Nov 04 02:54:43.520867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.520840 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca" pod "75df7aec-8910-4f5e-b60f-fe0239debfc3" (UID: "75df7aec-8910-4f5e-b60f-fe0239debfc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:43.521034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.521016 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:43.522817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.522786 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca" pod "75df7aec-8910-4f5e-b60f-fe0239debfc3" (UID: "75df7aec-8910-4f5e-b60f-fe0239debfc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:43.522817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.522797 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb" pod "75df7aec-8910-4f5e-b60f-fe0239debfc3" (UID: "75df7aec-8910-4f5e-b60f-fe0239debfc3"). InnerVolumeSpecName "kube-api-access-6m9kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:43.622410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.622373 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75df7aec-8910-4f5e-b60f-fe0239debfc3-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:43.622410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.622406 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9kb\" (UniqueName: \"kubernetes.io/projected/75df7aec-8910-4f5e-b60f-fe0239debfc3-kube-api-access-6m9kb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:43.987408 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.987336 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6"] Nov 04 02:54:43.988534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.988510 4784 cpu_manager.go:414] "containerMap: removing container" podUID="75df7aec-8910-4f5e-b60f-fe0239debfc3" containerName="revoke-keys" Nov 04 02:54:43.988534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.988533 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75df7aec-8910-4f5e-b60f-fe0239debfc3" containerName="revoke-keys" Nov 04 02:54:43.988702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.988697 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="75df7aec-8910-4f5e-b60f-fe0239debfc3" containerName="revoke-keys" Nov 04 02:54:43.993092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:43.993071 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.028510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.028468 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptvs\" (UniqueName: \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.028670 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.028517 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.028905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.028880 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.031688 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.031403 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6"] Nov 04 02:54:44.050540 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.050508 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:44.050697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.050564 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:44.130289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.130249 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptvs\" (UniqueName: \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.130289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.130295 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.130566 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.130489 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.131127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.131096 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.132905 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.132885 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.193106 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.193075 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptvs\" (UniqueName: \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\") pod \"maas-api-revoke-sub-90b77ddc-v8lj6\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.194250 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.194222 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" event={"ID":"75df7aec-8910-4f5e-b60f-fe0239debfc3","Type":"ContainerDied","Data":"19efb8ea61299a62bc775fb0a28b95df5089d3659d22d38a8474af66e22f08d1"} Nov 04 02:54:44.194388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.194280 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19efb8ea61299a62bc775fb0a28b95df5089d3659d22d38a8474af66e22f08d1" Nov 04 02:54:44.194388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.194281 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f" Nov 04 02:54:44.196310 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.196284 4784 generic.go:356] "Generic (PLEG): container finished" podID="383b1d22-3665-4ad5-912e-fb3afaf5f314" containerID="6e32c5cb21bf213ff518cfabaca836a9d6a6fe30420d42d911964ffb784b4103" exitCode=0 Nov 04 02:54:44.196423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.196324 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" event={"ID":"383b1d22-3665-4ad5-912e-fb3afaf5f314","Type":"ContainerDied","Data":"6e32c5cb21bf213ff518cfabaca836a9d6a6fe30420d42d911964ffb784b4103"} Nov 04 02:54:44.388610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.388524 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:44.790719 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:44.790690 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0543394_456d_4170_a13f_17f51cc6674f.slice/crio-552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2 WatchSource:0}: Error finding container 552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2: Status 404 returned error can't find the container with id 552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2 Nov 04 02:54:44.790719 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:44.791336 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6"] Nov 04 02:54:45.050174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.050087 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:45.050174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.050145 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:45.210567 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.210530 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" event={"ID":"d0543394-456d-4170-a13f-17f51cc6674f","Type":"ContainerStarted","Data":"552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2"} Nov 04 02:54:45.397604 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.397577 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:45.452578 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.452535 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\") pod \"383b1d22-3665-4ad5-912e-fb3afaf5f314\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " Nov 04 02:54:45.452771 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.452678 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\") pod \"383b1d22-3665-4ad5-912e-fb3afaf5f314\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " Nov 04 02:54:45.452833 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.452823 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\" (UniqueName: \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\") pod \"383b1d22-3665-4ad5-912e-fb3afaf5f314\" (UID: \"383b1d22-3665-4ad5-912e-fb3afaf5f314\") " Nov 04 02:54:45.453230 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.453043 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca" pod "383b1d22-3665-4ad5-912e-fb3afaf5f314" (UID: "383b1d22-3665-4ad5-912e-fb3afaf5f314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:45.453723 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.453700 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:45.455039 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.455007 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca" pod "383b1d22-3665-4ad5-912e-fb3afaf5f314" (UID: "383b1d22-3665-4ad5-912e-fb3afaf5f314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:45.455144 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.455119 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld" pod "383b1d22-3665-4ad5-912e-fb3afaf5f314" (UID: "383b1d22-3665-4ad5-912e-fb3afaf5f314"). InnerVolumeSpecName "kube-api-access-j75ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:45.556073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.555952 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75ld\" (UniqueName: \"kubernetes.io/projected/383b1d22-3665-4ad5-912e-fb3afaf5f314-kube-api-access-j75ld\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:45.556073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:45.556000 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b1d22-3665-4ad5-912e-fb3afaf5f314-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:46.051206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.051154 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 02:54:46.051421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.051291 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 02:54:46.222064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.221951 4784 generic.go:356] "Generic (PLEG): container finished" podID="d0543394-456d-4170-a13f-17f51cc6674f" containerID="61c55de286401eaa20d3c2d211e5a80820d3a8617de2b236355de00ba66b55b5" exitCode=0 Nov 04 02:54:46.222064 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.222045 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" event={"ID":"d0543394-456d-4170-a13f-17f51cc6674f","Type":"ContainerDied","Data":"61c55de286401eaa20d3c2d211e5a80820d3a8617de2b236355de00ba66b55b5"} Nov 04 02:54:46.224178 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.224153 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" Nov 04 02:54:46.224354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.224171 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx" event={"ID":"383b1d22-3665-4ad5-912e-fb3afaf5f314","Type":"ContainerDied","Data":"ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572"} Nov 04 02:54:46.224354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:46.224258 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0aba763f73559032bbc99cfa42c60a663304bff4b46ba83b85f2c0512d1572" Nov 04 02:54:47.056553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.056517 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:47.056914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.056882 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:47.057406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.057351 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6"] Nov 04 02:54:47.057726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.057703 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 02:54:47.064132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.064108 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l"] Nov 04 02:54:47.105519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.105489 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2aca13e-gdkw6"] Nov 04 02:54:47.111319 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.111282 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e199f9d2-fzb8l"] Nov 04 02:54:47.162936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.162904 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382f038a-26d0-4aff-8b61-c747cd78b678" path="/var/lib/kubelet/pods/382f038a-26d0-4aff-8b61-c747cd78b678/volumes" Nov 04 02:54:47.163491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.163475 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff3ac3f-eb19-494d-ac25-b5ec506ca21c" path="/var/lib/kubelet/pods/aff3ac3f-eb19-494d-ac25-b5ec506ca21c/volumes" Nov 04 02:54:47.450750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.450724 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:47.589171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.589135 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\" (UniqueName: \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\") pod \"d0543394-456d-4170-a13f-17f51cc6674f\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " Nov 04 02:54:47.589410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.589271 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\") pod \"d0543394-456d-4170-a13f-17f51cc6674f\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " Nov 04 02:54:47.589410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.589339 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\") pod \"d0543394-456d-4170-a13f-17f51cc6674f\" (UID: \"d0543394-456d-4170-a13f-17f51cc6674f\") " Nov 04 02:54:47.590006 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.589978 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca" pod "d0543394-456d-4170-a13f-17f51cc6674f" (UID: "d0543394-456d-4170-a13f-17f51cc6674f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:47.592071 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.592038 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca" pod "d0543394-456d-4170-a13f-17f51cc6674f" (UID: "d0543394-456d-4170-a13f-17f51cc6674f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:47.592720 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.592687 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs" pod "d0543394-456d-4170-a13f-17f51cc6674f" (UID: "d0543394-456d-4170-a13f-17f51cc6674f"). InnerVolumeSpecName "kube-api-access-kptvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:47.691361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.691329 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:47.691361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.691358 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0543394-456d-4170-a13f-17f51cc6674f-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:47.691361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.691369 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptvs\" (UniqueName: \"kubernetes.io/projected/d0543394-456d-4170-a13f-17f51cc6674f-kube-api-access-kptvs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:47.773669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.773630 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh"] Nov 04 02:54:47.774389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774339 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d0543394-456d-4170-a13f-17f51cc6674f" containerName="revoke-keys" Nov 04 02:54:47.774432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774390 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0543394-456d-4170-a13f-17f51cc6674f" containerName="revoke-keys" Nov 04 02:54:47.774491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774431 4784 cpu_manager.go:414] "containerMap: removing container" podUID="383b1d22-3665-4ad5-912e-fb3afaf5f314" containerName="revoke-keys" Nov 04 02:54:47.774491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774440 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383b1d22-3665-4ad5-912e-fb3afaf5f314" containerName="revoke-keys" Nov 04 02:54:47.774762 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774731 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0543394-456d-4170-a13f-17f51cc6674f" containerName="revoke-keys" Nov 04 02:54:47.774868 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.774767 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="383b1d22-3665-4ad5-912e-fb3afaf5f314" containerName="revoke-keys" Nov 04 02:54:47.778728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.778709 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.834915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.834880 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh"] Nov 04 02:54:47.894835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.894800 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.895041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.895020 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwclg\" (UniqueName: \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.895221 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.895172 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.997088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.996997 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.997088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.997082 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.997281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.997221 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwclg\" (UniqueName: \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.997802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.997777 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:47.999659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:47.999639 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:48.060897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.060855 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwclg\" (UniqueName: \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\") pod \"maas-api-revoke-sub-a3c99ad0-dpflh\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:48.094903 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.094866 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:48.243757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.243729 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" Nov 04 02:54:48.243929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.243763 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6" event={"ID":"d0543394-456d-4170-a13f-17f51cc6674f","Type":"ContainerDied","Data":"552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2"} Nov 04 02:54:48.243929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.243813 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552b884ee62fb32b91a8d3e1d1483ef55ed92cdb18480d076a555cbb549a48c2" Nov 04 02:54:48.481506 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:48.481476 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aa40cb_69e4_4eda_940b_180def10ab94.slice/crio-900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76 WatchSource:0}: Error finding container 900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76: Status 404 returned error can't find the container with id 900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76 Nov 04 02:54:48.483125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:48.483097 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh"] Nov 04 02:54:49.052419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.052329 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r"] Nov 04 02:54:49.057888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.057859 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a797974f-xl99r"] Nov 04 02:54:49.162934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.162893 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2559ba-8c67-4037-9486-0e5040e4a636" path="/var/lib/kubelet/pods/1d2559ba-8c67-4037-9486-0e5040e4a636/volumes" Nov 04 02:54:49.257920 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.257884 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" event={"ID":"02aa40cb-69e4-4eda-940b-180def10ab94","Type":"ContainerStarted","Data":"1828d0d1dd330db3499b4705fda7d7983b188d7b15705e29c31e680b26a1be3b"} Nov 04 02:54:49.258130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.257945 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" event={"ID":"02aa40cb-69e4-4eda-940b-180def10ab94","Type":"ContainerStarted","Data":"900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76"} Nov 04 02:54:49.376254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:49.376142 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" podStartSLOduration=2.376127672 podStartE2EDuration="2.376127672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:49.374997261 +0000 UTC m=+1336.341498253" watchObservedRunningTime="2026-11-04 02:54:49.376127672 +0000 UTC m=+1336.342628663" Nov 04 02:54:50.053492 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.053458 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d"] Nov 04 02:54:50.060334 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.060296 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad449bda-pxp5d"] Nov 04 02:54:50.080992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.080949 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49"] Nov 04 02:54:50.086835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.086798 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.163821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.163787 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49"] Nov 04 02:54:50.229720 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.229679 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9l7w\" (UniqueName: \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.230069 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.230043 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.230202 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.230135 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.268147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.268106 4784 generic.go:356] "Generic (PLEG): container finished" podID="02aa40cb-69e4-4eda-940b-180def10ab94" containerID="1828d0d1dd330db3499b4705fda7d7983b188d7b15705e29c31e680b26a1be3b" exitCode=0 Nov 04 02:54:50.268331 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.268221 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" event={"ID":"02aa40cb-69e4-4eda-940b-180def10ab94","Type":"ContainerDied","Data":"1828d0d1dd330db3499b4705fda7d7983b188d7b15705e29c31e680b26a1be3b"} Nov 04 02:54:50.331944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.331866 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.331944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.331921 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.332147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.331974 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9l7w\" (UniqueName: \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.332721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.332695 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.335823 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.335797 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.395792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.395761 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9l7w\" (UniqueName: \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\") pod \"maas-api-revoke-sub-18ac13b9-p6m49\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.410243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.410211 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:50.801259 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:50.801225 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7856e4f_cbd1_44b2_96f7_b76e64d778c5.slice/crio-5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18 WatchSource:0}: Error finding container 5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18: Status 404 returned error can't find the container with id 5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18 Nov 04 02:54:50.802879 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.802852 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49"] Nov 04 02:54:50.844070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.844027 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 02:54:50.853858 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.853823 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.870959 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.870936 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 02:54:50.875587 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.875568 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.913794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.906664 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948230 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948291 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948367 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948422 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948455 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948494 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948527 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948584 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgw4\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948611 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948634 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948723 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948772 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948862 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948919 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.948977 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbdz\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.949055 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:50.951510 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.949094 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:50.956904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:50.956854 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 02:54:51.050121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050022 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050085 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050157 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050258 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050308 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbdz\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050370 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050407 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050421 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050464 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050505 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050586 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050643 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050675 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050722 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.050897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050759 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050823 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgw4\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.050897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050849 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.051236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051213 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.051316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.050853 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.051316 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051283 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.051443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051326 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.051443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051357 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.051821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051798 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.051887 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051821 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.051950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.051917 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.052073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.052052 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.052140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.052079 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.052256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.052232 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.056400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.055937 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.056400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.056136 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.056400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.056368 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.056827 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.056444 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.116342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.116304 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.116486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.116411 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgw4\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.116486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.116422 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.116603 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.116587 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbdz\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.163437 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.163403 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c631659-850f-47ce-9e88-52ad342c750e" path="/var/lib/kubelet/pods/0c631659-850f-47ce-9e88-52ad342c750e/volumes" Nov 04 02:54:51.179848 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.179817 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:51.190077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.190045 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:51.290531 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.290501 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" event={"ID":"f7856e4f-cbd1-44b2-96f7-b76e64d778c5","Type":"ContainerStarted","Data":"5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18"} Nov 04 02:54:51.509480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.509454 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:51.623306 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:51.623271 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61210825_de7c_427c_87ca_ee7d63269346.slice/crio-6659581f8768dcca97d7689428b04dad7b18b508d6c55a14ccc7cbf2a3f6f147 WatchSource:0}: Error finding container 6659581f8768dcca97d7689428b04dad7b18b508d6c55a14ccc7cbf2a3f6f147: Status 404 returned error can't find the container with id 6659581f8768dcca97d7689428b04dad7b18b508d6c55a14ccc7cbf2a3f6f147 Nov 04 02:54:51.624970 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:54:51.624948 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09730a8d_9021_4166_ad4b_a72626adddbe.slice/crio-068f6e7ec83686704e42bda27d6f87508f0cd1896ff16e59b87ed5a0f249437b WatchSource:0}: Error finding container 068f6e7ec83686704e42bda27d6f87508f0cd1896ff16e59b87ed5a0f249437b: Status 404 returned error can't find the container with id 068f6e7ec83686704e42bda27d6f87508f0cd1896ff16e59b87ed5a0f249437b Nov 04 02:54:51.625769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.625693 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 02:54:51.626506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.626433 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.626607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.626559 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.626659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.626603 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.627571 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.627541 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.627655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.627612 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.627726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.627657 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:54:51.632106 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.632086 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 02:54:51.685741 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.685225 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\") pod \"02aa40cb-69e4-4eda-940b-180def10ab94\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " Nov 04 02:54:51.685741 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.685419 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\" (UniqueName: \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\") pod \"02aa40cb-69e4-4eda-940b-180def10ab94\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " Nov 04 02:54:51.685741 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.685534 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\") pod \"02aa40cb-69e4-4eda-940b-180def10ab94\" (UID: \"02aa40cb-69e4-4eda-940b-180def10ab94\") " Nov 04 02:54:51.690487 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.690442 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca" pod "02aa40cb-69e4-4eda-940b-180def10ab94" (UID: "02aa40cb-69e4-4eda-940b-180def10ab94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:51.690810 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.690773 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca" pod "02aa40cb-69e4-4eda-940b-180def10ab94" (UID: "02aa40cb-69e4-4eda-940b-180def10ab94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:51.708839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.708801 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg" pod "02aa40cb-69e4-4eda-940b-180def10ab94" (UID: "02aa40cb-69e4-4eda-940b-180def10ab94"). InnerVolumeSpecName "kube-api-access-dwclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:51.789927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.789883 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:51.790101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.789967 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwclg\" (UniqueName: \"kubernetes.io/projected/02aa40cb-69e4-4eda-940b-180def10ab94-kube-api-access-dwclg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:51.790101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:51.789989 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa40cb-69e4-4eda-940b-180def10ab94-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:52.317095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.317018 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" Nov 04 02:54:52.317095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.317044 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh" event={"ID":"02aa40cb-69e4-4eda-940b-180def10ab94","Type":"ContainerDied","Data":"900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76"} Nov 04 02:54:52.317512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.317105 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900fa09391abbeb3c6930310b719acbc668b1c2b682fd1689b16d90b634e9e76" Nov 04 02:54:52.319139 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.319116 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" event={"ID":"61210825-de7c-427c-87ca-ee7d63269346","Type":"ContainerStarted","Data":"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e"} Nov 04 02:54:52.319243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.319160 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" event={"ID":"61210825-de7c-427c-87ca-ee7d63269346","Type":"ContainerStarted","Data":"6659581f8768dcca97d7689428b04dad7b18b508d6c55a14ccc7cbf2a3f6f147"} Nov 04 02:54:52.321566 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.321543 4784 generic.go:356] "Generic (PLEG): container finished" podID="f7856e4f-cbd1-44b2-96f7-b76e64d778c5" containerID="9b96ad73177f7fcc61e99df3e01bd666d062dce8f5c2839b4fded5923fc47e31" exitCode=0 Nov 04 02:54:52.321680 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.321618 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" event={"ID":"f7856e4f-cbd1-44b2-96f7-b76e64d778c5","Type":"ContainerDied","Data":"9b96ad73177f7fcc61e99df3e01bd666d062dce8f5c2839b4fded5923fc47e31"} Nov 04 02:54:52.323856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.323263 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" event={"ID":"09730a8d-9021-4166-ad4b-a72626adddbe","Type":"ContainerStarted","Data":"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2"} Nov 04 02:54:52.323856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.323307 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" event={"ID":"09730a8d-9021-4166-ad4b-a72626adddbe","Type":"ContainerStarted","Data":"068f6e7ec83686704e42bda27d6f87508f0cd1896ff16e59b87ed5a0f249437b"} Nov 04 02:54:52.710261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.704286 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podStartSLOduration=2.70426599 podStartE2EDuration="2.70426599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:52.503766381 +0000 UTC m=+1339.470267375" watchObservedRunningTime="2026-11-04 02:54:52.70426599 +0000 UTC m=+1339.670766984" Nov 04 02:54:52.711816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:52.711752 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podStartSLOduration=2.711734208 podStartE2EDuration="2.711734208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:52.703985215 +0000 UTC m=+1339.670486208" watchObservedRunningTime="2026-11-04 02:54:52.711734208 +0000 UTC m=+1339.678235201" Nov 04 02:54:53.051834 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.051734 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp"] Nov 04 02:54:53.089239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.076790 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f60ae9e-5prkp"] Nov 04 02:54:53.210849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.210702 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c" path="/var/lib/kubelet/pods/88bbfdbc-9cbe-4858-8ff9-a6b69c3b832c/volumes" Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211223 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211238 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211265 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211276 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211288 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:53.211551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.211318 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:53.721460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.720412 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:53.729606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.729581 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\" (UniqueName: \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\") pod \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " Nov 04 02:54:53.729805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.729785 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\") pod \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " Nov 04 02:54:53.729989 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.729959 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\") pod \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\" (UID: \"f7856e4f-cbd1-44b2-96f7-b76e64d778c5\") " Nov 04 02:54:53.730307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.730285 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca" pod "f7856e4f-cbd1-44b2-96f7-b76e64d778c5" (UID: "f7856e4f-cbd1-44b2-96f7-b76e64d778c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:53.730558 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.730539 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:53.732243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.732215 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca" pod "f7856e4f-cbd1-44b2-96f7-b76e64d778c5" (UID: "f7856e4f-cbd1-44b2-96f7-b76e64d778c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:53.732494 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.732475 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w" pod "f7856e4f-cbd1-44b2-96f7-b76e64d778c5" (UID: "f7856e4f-cbd1-44b2-96f7-b76e64d778c5"). InnerVolumeSpecName "kube-api-access-q9l7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:53.831722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.831673 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9l7w\" (UniqueName: \"kubernetes.io/projected/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-kube-api-access-q9l7w\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:53.831722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:53.831702 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7856e4f-cbd1-44b2-96f7-b76e64d778c5-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:54.050964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.050928 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf"] Nov 04 02:54:54.080214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.079601 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a7f9e0f-bk9kf"] Nov 04 02:54:54.181214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.180597 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:54.181214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.180663 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:54.206218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.204609 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:54.206218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.204673 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:54.346256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.344548 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" Nov 04 02:54:54.346256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.344563 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49" event={"ID":"f7856e4f-cbd1-44b2-96f7-b76e64d778c5","Type":"ContainerDied","Data":"5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18"} Nov 04 02:54:54.346256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.344607 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc71c971b6ac726bd217c93d0e97e8e415f1dbe59ec65cdb85b8403e55e8e18" Nov 04 02:54:54.548148 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.548109 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb"] Nov 04 02:54:54.548871 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.548855 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f7856e4f-cbd1-44b2-96f7-b76e64d778c5" containerName="revoke-keys" Nov 04 02:54:54.548915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.548873 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7856e4f-cbd1-44b2-96f7-b76e64d778c5" containerName="revoke-keys" Nov 04 02:54:54.548952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.548921 4784 cpu_manager.go:414] "containerMap: removing container" podUID="02aa40cb-69e4-4eda-940b-180def10ab94" containerName="revoke-keys" Nov 04 02:54:54.548952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.548927 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02aa40cb-69e4-4eda-940b-180def10ab94" containerName="revoke-keys" Nov 04 02:54:54.549035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.549026 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="02aa40cb-69e4-4eda-940b-180def10ab94" containerName="revoke-keys" Nov 04 02:54:54.549070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.549040 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7856e4f-cbd1-44b2-96f7-b76e64d778c5" containerName="revoke-keys" Nov 04 02:54:54.552712 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.552696 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.610271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.607137 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:54:54.610271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.607231 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:54:54.610271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.607337 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:54:54.610271 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.607434 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:54:54.618086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.618017 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb"] Nov 04 02:54:54.713765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.713713 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.714968 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.713865 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.714968 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.714167 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcgr\" (UniqueName: \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.816753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.816708 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcgr\" (UniqueName: \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.817152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.816919 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.817152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.816963 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.818839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.818764 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.826024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.825622 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.902702 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.887498 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcgr\" (UniqueName: \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\") pod \"maas-api-revoke-sub-f493d43d-xn4hb\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:54.906691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:54.904737 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:55.198213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.191323 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:55.198213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.191407 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:55.198213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.192318 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:55.198213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.192372 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:55.229598 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.229560 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95341af-f780-44e8-86ed-97ec51dcd0ab" path="/var/lib/kubelet/pods/e95341af-f780-44e8-86ed-97ec51dcd0ab/volumes" Nov 04 02:54:55.338601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.338561 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z"] Nov 04 02:54:55.346483 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.346454 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z"] Nov 04 02:54:55.346716 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.346578 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.424415 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.424292 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb"] Nov 04 02:54:55.435501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.435468 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkh2p\" (UniqueName: \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.435687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.435675 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.435801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.435791 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.537599 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.537567 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkh2p\" (UniqueName: \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.537768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.537617 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.537768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.537654 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.538509 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.538418 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.540305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.540285 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.611630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.611588 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkh2p\" (UniqueName: \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\") pod \"maas-api-revoke-sub-6fc8f674-72n8z\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:55.754710 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:55.754632 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:56.205215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.196705 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:56.205215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.196777 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:56.205215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.197769 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:56.205215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.197816 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:56.285270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.285135 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z"] Nov 04 02:54:56.420293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.420254 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" event={"ID":"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a","Type":"ContainerStarted","Data":"b929e2de57d604806013a657c8ff87169ee45cf772e6bbaade496783e84be323"} Nov 04 02:54:56.420601 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.420575 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" event={"ID":"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a","Type":"ContainerStarted","Data":"364b0afd08d952210eefbd26a29d8e18ce98c34004de326de07beaee2f9df6ac"} Nov 04 02:54:56.422328 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.422304 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" event={"ID":"c103f880-81b9-4174-81cb-30f8cac0245e","Type":"ContainerStarted","Data":"70194fab26fe8f207a073272715826588ffbd7cc3a07decebbeecc3e9701bfba"} Nov 04 02:54:56.540853 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:56.540746 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" podStartSLOduration=2.540732187 podStartE2EDuration="2.540732187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:56.538296796 +0000 UTC m=+1343.504797786" watchObservedRunningTime="2026-11-04 02:54:56.540732187 +0000 UTC m=+1343.507233215" Nov 04 02:54:57.221153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.220626 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:57.221153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.220698 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:57.221153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.221080 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:57.221153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.221136 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:57.434742 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.434707 4784 generic.go:356] "Generic (PLEG): container finished" podID="8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" containerID="b929e2de57d604806013a657c8ff87169ee45cf772e6bbaade496783e84be323" exitCode=0 Nov 04 02:54:57.434930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.434751 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" event={"ID":"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a","Type":"ContainerDied","Data":"b929e2de57d604806013a657c8ff87169ee45cf772e6bbaade496783e84be323"} Nov 04 02:54:57.436799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.436779 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" event={"ID":"c103f880-81b9-4174-81cb-30f8cac0245e","Type":"ContainerStarted","Data":"569aaab9898221af44895da41b7bbd8f3887cbcbadd9e88baf780df1687aa04a"} Nov 04 02:54:57.720214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:57.719865 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" podStartSLOduration=2.719842895 podStartE2EDuration="2.719842895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:57.677732322 +0000 UTC m=+1344.644233319" watchObservedRunningTime="2026-11-04 02:54:57.719842895 +0000 UTC m=+1344.686343893" Nov 04 02:54:58.052041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.052002 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5"] Nov 04 02:54:58.058255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.058216 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c37478-szlz5"] Nov 04 02:54:58.180176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.180140 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 02:54:58.180418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.180217 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 02:54:58.191292 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.191254 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 02:54:58.191463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.191327 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 02:54:58.450707 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.450675 4784 generic.go:356] "Generic (PLEG): container finished" podID="c103f880-81b9-4174-81cb-30f8cac0245e" containerID="569aaab9898221af44895da41b7bbd8f3887cbcbadd9e88baf780df1687aa04a" exitCode=0 Nov 04 02:54:58.451233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.450779 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" event={"ID":"c103f880-81b9-4174-81cb-30f8cac0245e","Type":"ContainerDied","Data":"569aaab9898221af44895da41b7bbd8f3887cbcbadd9e88baf780df1687aa04a"} Nov 04 02:54:58.708850 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.708780 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:58.849573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.849532 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\") pod \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " Nov 04 02:54:58.849765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.849604 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\" (UniqueName: \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\") pod \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " Nov 04 02:54:58.849765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.849696 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\") pod \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\" (UID: \"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a\") " Nov 04 02:54:58.853040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.850859 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca" pod "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" (UID: "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:58.862206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.853326 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr" pod "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" (UID: "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a"). InnerVolumeSpecName "kube-api-access-vxcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:58.862206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.853482 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca" pod "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" (UID: "8280623a-b1b0-473f-9ef7-7b13c9ce2c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:58.952811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.952770 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:58.952811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.952805 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxcgr\" (UniqueName: \"kubernetes.io/projected/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-kube-api-access-vxcgr\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:58.952811 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:58.952821 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:59.169760 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.169717 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3613e1-a4ff-4a7d-81ce-417cc2def46e" path="/var/lib/kubelet/pods/2d3613e1-a4ff-4a7d-81ce-417cc2def46e/volumes" Nov 04 02:54:59.452435 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.452353 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:59.453295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.453266 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:59.454178 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.454158 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 02:54:59.465981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.465946 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" Nov 04 02:54:59.465981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.465967 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb" event={"ID":"8280623a-b1b0-473f-9ef7-7b13c9ce2c7a","Type":"ContainerDied","Data":"364b0afd08d952210eefbd26a29d8e18ce98c34004de326de07beaee2f9df6ac"} Nov 04 02:54:59.466181 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.466029 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364b0afd08d952210eefbd26a29d8e18ce98c34004de326de07beaee2f9df6ac" Nov 04 02:54:59.534840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.534806 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:59.536236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.536211 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:59.537404 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.537385 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 02:54:59.706447 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.706376 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:54:59.767842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.767792 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\" (UniqueName: \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\") pod \"c103f880-81b9-4174-81cb-30f8cac0245e\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " Nov 04 02:54:59.768041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.768026 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\") pod \"c103f880-81b9-4174-81cb-30f8cac0245e\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " Nov 04 02:54:59.768252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.768233 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\") pod \"c103f880-81b9-4174-81cb-30f8cac0245e\" (UID: \"c103f880-81b9-4174-81cb-30f8cac0245e\") " Nov 04 02:54:59.768641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.768608 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca" pod "c103f880-81b9-4174-81cb-30f8cac0245e" (UID: "c103f880-81b9-4174-81cb-30f8cac0245e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:54:59.769079 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.769056 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:59.770465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.770439 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca" pod "c103f880-81b9-4174-81cb-30f8cac0245e" (UID: "c103f880-81b9-4174-81cb-30f8cac0245e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:54:59.770698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.770674 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p" pod "c103f880-81b9-4174-81cb-30f8cac0245e" (UID: "c103f880-81b9-4174-81cb-30f8cac0245e"). InnerVolumeSpecName "kube-api-access-fkh2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:54:59.870668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.870638 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c103f880-81b9-4174-81cb-30f8cac0245e-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:54:59.870668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:54:59.870662 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkh2p\" (UniqueName: \"kubernetes.io/projected/c103f880-81b9-4174-81cb-30f8cac0245e-kube-api-access-fkh2p\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:00.059626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:00.059444 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4"] Nov 04 02:55:00.067783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:00.067745 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b86fb9a-rc2z4"] Nov 04 02:55:00.479174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:00.479140 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" Nov 04 02:55:00.479174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:00.479144 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z" event={"ID":"c103f880-81b9-4174-81cb-30f8cac0245e","Type":"ContainerDied","Data":"70194fab26fe8f207a073272715826588ffbd7cc3a07decebbeecc3e9701bfba"} Nov 04 02:55:00.479673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:00.479248 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70194fab26fe8f207a073272715826588ffbd7cc3a07decebbeecc3e9701bfba" Nov 04 02:55:01.163823 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:01.163788 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdec99c5-31d3-408a-84e9-2b5a37490278" path="/var/lib/kubelet/pods/fdec99c5-31d3-408a-84e9-2b5a37490278/volumes" Nov 04 02:55:04.015228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.011528 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.102216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.101352 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.415456 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.415412 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:04.521289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.518297 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.792290 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.790291 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.817235 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.812307 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.852220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.849334 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.884423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:04.874279 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.021217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.020306 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.021217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.020802 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:55:05.050282 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.029857 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.060485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.060438 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.114165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.114120 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:05.115149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.115124 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:55:05.440281 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.433295 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.528344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.528300 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.656549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.621570 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:55:05.656901 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.656872 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:55:05.845276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.820605 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.858414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.824337 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:05.906605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.894304 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:05.906605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.898782 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:55:05.906605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.898870 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:55:05.906605 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.898930 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:55:05.950228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.930331 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.950228 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:05.932744 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:55:06.086955 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.086909 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:06.088799 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.088765 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:06.131275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.115377 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:55:06.131275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.115450 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:55:06.131275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.117771 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.131275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.130275 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:06.535366 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.535303 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.731216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.695814 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.843488 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:06.843377 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:55:06.869242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.858255 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.869242 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.864696 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:06.906303 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.906235 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.932528 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:06.932484 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:55:07.135543 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.129873 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.155401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.155353 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.225216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.157253 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.225216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.157587 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.387490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.387286 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:07.387490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.387361 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.552218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.544292 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:55:07.959215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.841511 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:07.996227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.870395 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:08.015227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.872034 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.192038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.953298 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:08.192038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:07.958818 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:55:08.263307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.259996 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.263307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.260133 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:08.445591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.445541 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:08.712222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.710298 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:08.746219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.739070 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:08.828220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.762653 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 04 02:55:08.959422 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.959377 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc"] Nov 04 02:55:09.018602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.960496 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:09.018989 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.961713 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 02:55:09.019209 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.961738 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.019562 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.988319 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": context deadline exceeded" Nov 04 02:55:09.019746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:08.997156 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.078219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.052924 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaffa84c-8c6sc"] Nov 04 02:55:09.100764 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.058316 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.119728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.058360 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:09.119940 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.058381 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:09.284666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.270353 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.345975 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.345933 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3735aa5e-9240-4e83-a052-e6b90826d98e" path="/var/lib/kubelet/pods/3735aa5e-9240-4e83-a052-e6b90826d98e/volumes" Nov 04 02:55:09.457217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.451291 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.715213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.711447 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.749280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.749243 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4"] Nov 04 02:55:09.766458 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.766420 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c103f880-81b9-4174-81cb-30f8cac0245e" containerName="revoke-keys" Nov 04 02:55:09.766701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.766681 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c103f880-81b9-4174-81cb-30f8cac0245e" containerName="revoke-keys" Nov 04 02:55:09.766823 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.766809 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" containerName="revoke-keys" Nov 04 02:55:09.766927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.766914 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" containerName="revoke-keys" Nov 04 02:55:09.771632 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.771606 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c103f880-81b9-4174-81cb-30f8cac0245e" containerName="revoke-keys" Nov 04 02:55:09.795627 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.795598 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" containerName="revoke-keys" Nov 04 02:55:09.797819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.794961 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:09.850842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.850808 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4"] Nov 04 02:55:09.869982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.869345 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:09.934216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.928827 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:55:09.934216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.929220 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:55:09.934216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.929335 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:55:09.938930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.938885 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:09.939436 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.939408 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hw4\" (UniqueName: \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:09.939621 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.939602 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:09.954342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.954300 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:09.954625 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.954594 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.990011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:09.989509 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:55:10.050072 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.050033 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:10.076180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.050470 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.076180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.050767 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.077887 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.078064 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hw4\" (UniqueName: \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.078102 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.079141 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.102283 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.126635 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:10.142794 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.127043 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.286309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.153258 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.286309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.274298 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.369256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.311016 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hw4\" (UniqueName: \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\") pod \"maas-api-revoke-sub-a45fd7b5-jvnk4\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.584818 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.501698 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.584818 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.547533 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:10.667341 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.667281 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded" Nov 04 02:55:10.711807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.707295 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:10.711807 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.707365 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.738361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.712575 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:10.816646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.814358 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.816646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.814689 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:10.923408 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.923366 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:10.923713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:10.923677 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.091215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.061348 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.091215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.089299 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:55:11.091215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.089929 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:11.249824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.128578 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:11.249824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.128977 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:11.249824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.130756 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.276333 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.276281 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.416349 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.416304 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:11.510279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.500012 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.510279 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.503988 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:11.974248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.964556 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:11.974248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:11.966898 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:12.082217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.073011 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.103691 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.093452 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:12.163252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.159814 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.198797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.198750 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:55:12.231275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.210436 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:12.345217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.223349 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.838573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.205856 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 04 02:55:12.843293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.230518 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Nov 04 02:55:12.919209 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.740330 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:12.961223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.831491 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:55:12.961223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.958084 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4"] Nov 04 02:55:12.961223 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.910683 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:12.967371 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.967339 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:55:12.982362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.982335 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:55:12.999603 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.999533 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:12.999754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:12.999708 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:13.013249 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.009264 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" event={"ID":"73054940-7c23-43aa-b67f-8baa65c9a2f8","Type":"ContainerStarted","Data":"8b83558b6fdecd434b89d3a1b68988e97fbd364a42967ddcbc88904148e8144e"} Nov 04 02:55:13.097886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.097256 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:55:13.097886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.097323 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:55:13.097886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.097356 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:55:13.098322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.098302 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:55:13.098573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.098558 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:55:13.116288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.115099 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:55:13.116288 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.115181 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:55:13.536405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.536365 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w"] Nov 04 02:55:13.542442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.542414 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.575610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.575579 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm"] Nov 04 02:55:13.580522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.580499 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.597684 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.597655 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w"] Nov 04 02:55:13.616354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616324 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2tl\" (UniqueName: \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.616523 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616391 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.616564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616541 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.616723 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616698 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.616814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616796 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.616933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.616917 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfjp\" (UniqueName: \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.672837 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.672796 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm"] Nov 04 02:55:13.718522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718489 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.718721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718573 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfjp\" (UniqueName: \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.718721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718647 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2tl\" (UniqueName: \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.718721 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718686 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.718883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718785 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.718940 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.718898 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.719760 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.719735 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.719945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.719921 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.721803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.721776 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.722050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.722026 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.782867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.782836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2tl\" (UniqueName: \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\") pod \"maas-api-revoke-sub-bf1b5482-r8g9w\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.783053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.783032 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfjp\" (UniqueName: \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\") pod \"maas-api-revoke-sub-015f46a4-7flpm\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:13.865709 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.865676 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:13.898992 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:13.898960 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:14.028886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:14.028795 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" event={"ID":"73054940-7c23-43aa-b67f-8baa65c9a2f8","Type":"ContainerStarted","Data":"fe84aa48ae3afd50cd4627a4099238ed37f0a35e5dbbbb2b0b7a6e100b798634"} Nov 04 02:55:14.273806 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:14.273760 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc084feb0_7589_49da_858a_b750a9424bb9.slice/crio-f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4 WatchSource:0}: Error finding container f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4: Status 404 returned error can't find the container with id f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4 Nov 04 02:55:14.276110 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:14.276090 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w"] Nov 04 02:55:14.442557 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:14.442527 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm"] Nov 04 02:55:14.460870 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:14.460806 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a3a56e_1335_450d_8288_6ceefa517109.slice/crio-d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e WatchSource:0}: Error finding container d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e: Status 404 returned error can't find the container with id d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e Nov 04 02:55:14.492381 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:14.492319 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" podStartSLOduration=6.492299862 podStartE2EDuration="6.492299862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:14.490447509 +0000 UTC m=+1361.456948502" watchObservedRunningTime="2026-11-04 02:55:14.492299862 +0000 UTC m=+1361.458800854" Nov 04 02:55:15.059210 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.059100 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" event={"ID":"51a3a56e-1335-450d-8288-6ceefa517109","Type":"ContainerStarted","Data":"584b026aa1a4bbae50786b5ea1f7260d57fc854f6cba276b7e415a51e2e0bee0"} Nov 04 02:55:15.059661 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.059232 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" event={"ID":"51a3a56e-1335-450d-8288-6ceefa517109","Type":"ContainerStarted","Data":"d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e"} Nov 04 02:55:15.060403 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.060380 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf"] Nov 04 02:55:15.061464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.061440 4784 generic.go:356] "Generic (PLEG): container finished" podID="73054940-7c23-43aa-b67f-8baa65c9a2f8" containerID="fe84aa48ae3afd50cd4627a4099238ed37f0a35e5dbbbb2b0b7a6e100b798634" exitCode=0 Nov 04 02:55:15.061609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.061520 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" event={"ID":"73054940-7c23-43aa-b67f-8baa65c9a2f8","Type":"ContainerDied","Data":"fe84aa48ae3afd50cd4627a4099238ed37f0a35e5dbbbb2b0b7a6e100b798634"} Nov 04 02:55:15.063506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.063471 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" event={"ID":"c084feb0-7589-49da-858a-b750a9424bb9","Type":"ContainerStarted","Data":"b6cff63b70a9c0cb8feb5c429688f95560c3c0f95e0f45dc561d02d1d99a946e"} Nov 04 02:55:15.063616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.063535 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" event={"ID":"c084feb0-7589-49da-858a-b750a9424bb9","Type":"ContainerStarted","Data":"f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4"} Nov 04 02:55:15.066350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.066324 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06f7523-8klrf"] Nov 04 02:55:15.175693 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.175660 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e757a7-15ab-48e8-a173-f5860fbaa43b" path="/var/lib/kubelet/pods/54e757a7-15ab-48e8-a173-f5860fbaa43b/volumes" Nov 04 02:55:15.180656 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.180608 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" podStartSLOduration=2.180593651 podStartE2EDuration="2.180593651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:15.176914959 +0000 UTC m=+1362.143415955" watchObservedRunningTime="2026-11-04 02:55:15.180593651 +0000 UTC m=+1362.147094693" Nov 04 02:55:15.283182 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.283144 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq"] Nov 04 02:55:15.288365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.288338 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.299959 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.299907 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" podStartSLOduration=2.299890094 podStartE2EDuration="2.299890094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:15.299506164 +0000 UTC m=+1362.266007155" watchObservedRunningTime="2026-11-04 02:55:15.299890094 +0000 UTC m=+1362.266391089" Nov 04 02:55:15.345470 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.345435 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq"] Nov 04 02:55:15.358077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.358051 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.358552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.358114 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwls\" (UniqueName: \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.358653 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.358560 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.460409 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.460372 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.460557 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.460518 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.460632 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.460594 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwls\" (UniqueName: \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.461394 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.461367 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.463105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.463080 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.523118 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.523082 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwls\" (UniqueName: \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\") pod \"maas-api-revoke-sub-d59d253f-8kzgq\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.604842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.604757 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:15.990062 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:15.990033 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d352ef6_2d6e_4854_b9a8_19fd2686f641.slice/crio-8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd WatchSource:0}: Error finding container 8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd: Status 404 returned error can't find the container with id 8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd Nov 04 02:55:15.992174 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:15.992144 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq"] Nov 04 02:55:16.074518 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.074486 4784 generic.go:356] "Generic (PLEG): container finished" podID="51a3a56e-1335-450d-8288-6ceefa517109" containerID="584b026aa1a4bbae50786b5ea1f7260d57fc854f6cba276b7e415a51e2e0bee0" exitCode=0 Nov 04 02:55:16.074925 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.074579 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" event={"ID":"51a3a56e-1335-450d-8288-6ceefa517109","Type":"ContainerDied","Data":"584b026aa1a4bbae50786b5ea1f7260d57fc854f6cba276b7e415a51e2e0bee0"} Nov 04 02:55:16.076101 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.076079 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" event={"ID":"5d352ef6-2d6e-4854-b9a8-19fd2686f641","Type":"ContainerStarted","Data":"8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd"} Nov 04 02:55:16.077608 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.077586 4784 generic.go:356] "Generic (PLEG): container finished" podID="c084feb0-7589-49da-858a-b750a9424bb9" containerID="b6cff63b70a9c0cb8feb5c429688f95560c3c0f95e0f45dc561d02d1d99a946e" exitCode=0 Nov 04 02:55:16.077708 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.077636 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" event={"ID":"c084feb0-7589-49da-858a-b750a9424bb9","Type":"ContainerDied","Data":"b6cff63b70a9c0cb8feb5c429688f95560c3c0f95e0f45dc561d02d1d99a946e"} Nov 04 02:55:16.265147 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.265115 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:16.376519 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.376481 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\") pod \"73054940-7c23-43aa-b67f-8baa65c9a2f8\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " Nov 04 02:55:16.376651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.376557 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\") pod \"73054940-7c23-43aa-b67f-8baa65c9a2f8\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " Nov 04 02:55:16.376651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.376606 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\" (UniqueName: \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\") pod \"73054940-7c23-43aa-b67f-8baa65c9a2f8\" (UID: \"73054940-7c23-43aa-b67f-8baa65c9a2f8\") " Nov 04 02:55:16.376929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.376900 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca" pod "73054940-7c23-43aa-b67f-8baa65c9a2f8" (UID: "73054940-7c23-43aa-b67f-8baa65c9a2f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:16.377492 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.377470 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:16.378804 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.378776 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca" pod "73054940-7c23-43aa-b67f-8baa65c9a2f8" (UID: "73054940-7c23-43aa-b67f-8baa65c9a2f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:16.378926 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.378897 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4" pod "73054940-7c23-43aa-b67f-8baa65c9a2f8" (UID: "73054940-7c23-43aa-b67f-8baa65c9a2f8"). InnerVolumeSpecName "kube-api-access-c8hw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:16.479869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.479834 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73054940-7c23-43aa-b67f-8baa65c9a2f8-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:16.479869 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:16.479867 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8hw4\" (UniqueName: \"kubernetes.io/projected/73054940-7c23-43aa-b67f-8baa65c9a2f8-kube-api-access-c8hw4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.106603 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.106448 4784 generic.go:356] "Generic (PLEG): container finished" podID="5d352ef6-2d6e-4854-b9a8-19fd2686f641" containerID="dbed71d1b30ab12907a531f2d4c71e4a661438c52c0c3eb7d023061e4c035ee5" exitCode=0 Nov 04 02:55:17.107251 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.107221 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" event={"ID":"5d352ef6-2d6e-4854-b9a8-19fd2686f641","Type":"ContainerDied","Data":"dbed71d1b30ab12907a531f2d4c71e4a661438c52c0c3eb7d023061e4c035ee5"} Nov 04 02:55:17.113740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.113711 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" Nov 04 02:55:17.136212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.125618 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4" event={"ID":"73054940-7c23-43aa-b67f-8baa65c9a2f8","Type":"ContainerDied","Data":"8b83558b6fdecd434b89d3a1b68988e97fbd364a42967ddcbc88904148e8144e"} Nov 04 02:55:17.136212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.125680 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b83558b6fdecd434b89d3a1b68988e97fbd364a42967ddcbc88904148e8144e" Nov 04 02:55:17.416209 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.416155 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:17.485843 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.485812 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:17.508884 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.508846 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\" (UniqueName: \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\") pod \"51a3a56e-1335-450d-8288-6ceefa517109\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " Nov 04 02:55:17.509073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.508915 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\") pod \"51a3a56e-1335-450d-8288-6ceefa517109\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " Nov 04 02:55:17.509073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.508961 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\") pod \"51a3a56e-1335-450d-8288-6ceefa517109\" (UID: \"51a3a56e-1335-450d-8288-6ceefa517109\") " Nov 04 02:55:17.509352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.509316 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca" pod "51a3a56e-1335-450d-8288-6ceefa517109" (UID: "51a3a56e-1335-450d-8288-6ceefa517109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:17.509724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.509703 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.511338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.511305 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca" pod "51a3a56e-1335-450d-8288-6ceefa517109" (UID: "51a3a56e-1335-450d-8288-6ceefa517109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:17.511615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.511599 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp" pod "51a3a56e-1335-450d-8288-6ceefa517109" (UID: "51a3a56e-1335-450d-8288-6ceefa517109"). InnerVolumeSpecName "kube-api-access-jnfjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.610875 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\" (UniqueName: \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\") pod \"c084feb0-7589-49da-858a-b750a9424bb9\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.610977 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\") pod \"c084feb0-7589-49da-858a-b750a9424bb9\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.611077 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\") pod \"c084feb0-7589-49da-858a-b750a9424bb9\" (UID: \"c084feb0-7589-49da-858a-b750a9424bb9\") " Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.611945 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a3a56e-1335-450d-8288-6ceefa517109-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.611970 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnfjp\" (UniqueName: \"kubernetes.io/projected/51a3a56e-1335-450d-8288-6ceefa517109-kube-api-access-jnfjp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.616211 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.612383 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca" pod "c084feb0-7589-49da-858a-b750a9424bb9" (UID: "c084feb0-7589-49da-858a-b750a9424bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:17.630631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.630590 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl" pod "c084feb0-7589-49da-858a-b750a9424bb9" (UID: "c084feb0-7589-49da-858a-b750a9424bb9"). InnerVolumeSpecName "kube-api-access-rz2tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:17.633132 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.633056 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca" pod "c084feb0-7589-49da-858a-b750a9424bb9" (UID: "c084feb0-7589-49da-858a-b750a9424bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:17.714029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.713987 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2tl\" (UniqueName: \"kubernetes.io/projected/c084feb0-7589-49da-858a-b750a9424bb9-kube-api-access-rz2tl\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.714029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.714027 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:17.714355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:17.714043 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084feb0-7589-49da-858a-b750a9424bb9-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:18.128778 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.128748 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" Nov 04 02:55:18.128778 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.128753 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w" event={"ID":"c084feb0-7589-49da-858a-b750a9424bb9","Type":"ContainerDied","Data":"f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4"} Nov 04 02:55:18.129396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.128810 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b93726e6dab1794def4a3e2e06e7c7d10fbdd665c37c0eee0b24741d247ce4" Nov 04 02:55:18.130930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.130905 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" Nov 04 02:55:18.131081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.131063 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm" event={"ID":"51a3a56e-1335-450d-8288-6ceefa517109","Type":"ContainerDied","Data":"d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e"} Nov 04 02:55:18.131141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.131110 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d15296e3a73b522c4d17b69f6237918b09daaa90e0a4ec6884a7d477aab90f5e" Nov 04 02:55:18.369779 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.369754 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:18.443787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.443754 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\") pod \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " Nov 04 02:55:18.444020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.443904 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\" (UniqueName: \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\") pod \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " Nov 04 02:55:18.444020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.443978 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\") pod \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\" (UID: \"5d352ef6-2d6e-4854-b9a8-19fd2686f641\") " Nov 04 02:55:18.444463 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.444436 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca" pod "5d352ef6-2d6e-4854-b9a8-19fd2686f641" (UID: "5d352ef6-2d6e-4854-b9a8-19fd2686f641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:18.444892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.444865 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:18.446181 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.446155 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca" pod "5d352ef6-2d6e-4854-b9a8-19fd2686f641" (UID: "5d352ef6-2d6e-4854-b9a8-19fd2686f641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:18.446303 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.446279 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls" pod "5d352ef6-2d6e-4854-b9a8-19fd2686f641" (UID: "5d352ef6-2d6e-4854-b9a8-19fd2686f641"). InnerVolumeSpecName "kube-api-access-kgwls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:18.546330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.546290 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d352ef6-2d6e-4854-b9a8-19fd2686f641-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:18.546330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.546328 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgwls\" (UniqueName: \"kubernetes.io/projected/5d352ef6-2d6e-4854-b9a8-19fd2686f641-kube-api-access-kgwls\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:18.623452 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.623410 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d"] Nov 04 02:55:18.624560 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624539 4784 cpu_manager.go:414] "containerMap: removing container" podUID="73054940-7c23-43aa-b67f-8baa65c9a2f8" containerName="revoke-keys" Nov 04 02:55:18.624609 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624566 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73054940-7c23-43aa-b67f-8baa65c9a2f8" containerName="revoke-keys" Nov 04 02:55:18.624706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624629 4784 cpu_manager.go:414] "containerMap: removing container" podUID="5d352ef6-2d6e-4854-b9a8-19fd2686f641" containerName="revoke-keys" Nov 04 02:55:18.624706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624638 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d352ef6-2d6e-4854-b9a8-19fd2686f641" containerName="revoke-keys" Nov 04 02:55:18.624706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624670 4784 cpu_manager.go:414] "containerMap: removing container" podUID="51a3a56e-1335-450d-8288-6ceefa517109" containerName="revoke-keys" Nov 04 02:55:18.624706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624678 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a3a56e-1335-450d-8288-6ceefa517109" containerName="revoke-keys" Nov 04 02:55:18.624706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624700 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c084feb0-7589-49da-858a-b750a9424bb9" containerName="revoke-keys" Nov 04 02:55:18.624894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624707 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c084feb0-7589-49da-858a-b750a9424bb9" containerName="revoke-keys" Nov 04 02:55:18.624894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624890 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a3a56e-1335-450d-8288-6ceefa517109" containerName="revoke-keys" Nov 04 02:55:18.624964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624906 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c084feb0-7589-49da-858a-b750a9424bb9" containerName="revoke-keys" Nov 04 02:55:18.624964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624919 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="73054940-7c23-43aa-b67f-8baa65c9a2f8" containerName="revoke-keys" Nov 04 02:55:18.624964 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.624936 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d352ef6-2d6e-4854-b9a8-19fd2686f641" containerName="revoke-keys" Nov 04 02:55:18.629732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.629710 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.684350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.684318 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d"] Nov 04 02:55:18.760860 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.760772 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.760860 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.760809 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54x7\" (UniqueName: \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.761067 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.760969 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.862934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.862895 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.863120 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.863107 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.863167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.863127 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54x7\" (UniqueName: \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.863796 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.863767 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.865559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.865538 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.925421 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.925393 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54x7\" (UniqueName: \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\") pod \"maas-api-revoke-sub-5ef3b302-j7l2d\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:18.946091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:18.946054 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:19.142812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:19.142779 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" Nov 04 02:55:19.142812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:19.142797 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq" event={"ID":"5d352ef6-2d6e-4854-b9a8-19fd2686f641","Type":"ContainerDied","Data":"8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd"} Nov 04 02:55:19.143384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:19.142873 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a104d89f9eb58c58c3f8b2c2f2289c5680aacab522b81252ad536b72b3187dd" Nov 04 02:55:19.330639 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:19.330598 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555c8593_a5dc_4142_bd3d_5a700673f12a.slice/crio-d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66 WatchSource:0}: Error finding container d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66: Status 404 returned error can't find the container with id d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66 Nov 04 02:55:19.333529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:19.333505 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d"] Nov 04 02:55:20.157736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:20.157518 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" event={"ID":"555c8593-a5dc-4142-bd3d-5a700673f12a","Type":"ContainerStarted","Data":"ed53aa66bdcef30ce7ac9627dc1b05a5f3d172ce3dc10af547c267a8d7f9df37"} Nov 04 02:55:20.157736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:20.157578 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" event={"ID":"555c8593-a5dc-4142-bd3d-5a700673f12a","Type":"ContainerStarted","Data":"d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66"} Nov 04 02:55:20.277066 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:20.277011 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" podStartSLOduration=2.276996649 podStartE2EDuration="2.276996649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:20.275407221 +0000 UTC m=+1367.241908245" watchObservedRunningTime="2026-11-04 02:55:20.276996649 +0000 UTC m=+1367.243497642" Nov 04 02:55:21.005501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.005465 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 02:55:21.011499 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.011472 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103019 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.102981 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103044 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103073 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103219 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103280 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103366 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5v4\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103430 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103592 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.103671 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.103644 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.123318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.123276 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 02:55:21.171845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.171810 4784 generic.go:356] "Generic (PLEG): container finished" podID="555c8593-a5dc-4142-bd3d-5a700673f12a" containerID="ed53aa66bdcef30ce7ac9627dc1b05a5f3d172ce3dc10af547c267a8d7f9df37" exitCode=0 Nov 04 02:55:21.172249 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.171891 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" event={"ID":"555c8593-a5dc-4142-bd3d-5a700673f12a","Type":"ContainerDied","Data":"ed53aa66bdcef30ce7ac9627dc1b05a5f3d172ce3dc10af547c267a8d7f9df37"} Nov 04 02:55:21.204645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204594 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.204645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204643 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.204904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204712 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5v4\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.204904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204774 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.204904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204867 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205070 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.204911 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205228 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205284 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205291 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205358 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205314 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205582 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205743 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.205770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.205761 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.206164 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.206134 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.207918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.207888 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.208039 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.208022 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.267673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.267592 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.267673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.267648 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5v4\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.326839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.326806 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:21.773408 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:21.773374 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f62f36_e794_4802_aa54_d2a93106fd9e.slice/crio-9ae60dbf42e7dae0aa8b0979104c29d78d4a0168649a6b1c97bf282d1c1d12d3 WatchSource:0}: Error finding container 9ae60dbf42e7dae0aa8b0979104c29d78d4a0168649a6b1c97bf282d1c1d12d3: Status 404 returned error can't find the container with id 9ae60dbf42e7dae0aa8b0979104c29d78d4a0168649a6b1c97bf282d1c1d12d3 Nov 04 02:55:21.775032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.775003 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 02:55:21.776298 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.776265 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:21.776377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.776334 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:21.776377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:21.776363 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:22.059464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.059376 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t"] Nov 04 02:55:22.067058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.067028 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc247602-l2h5t"] Nov 04 02:55:22.185969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.185933 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" event={"ID":"08f62f36-e794-4802-aa54-d2a93106fd9e","Type":"ContainerStarted","Data":"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf"} Nov 04 02:55:22.186465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.185988 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" event={"ID":"08f62f36-e794-4802-aa54-d2a93106fd9e","Type":"ContainerStarted","Data":"9ae60dbf42e7dae0aa8b0979104c29d78d4a0168649a6b1c97bf282d1c1d12d3"} Nov 04 02:55:22.334607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.334511 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:22.335890 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.335851 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:22.336052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.335918 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:22.340976 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.340922 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podStartSLOduration=2.340903505 podStartE2EDuration="2.340903505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:22.313772248 +0000 UTC m=+1369.280273253" watchObservedRunningTime="2026-11-04 02:55:22.340903505 +0000 UTC m=+1369.307404498" Nov 04 02:55:22.402152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.402090 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:22.424897 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.424866 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\" (UniqueName: \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\") pod \"555c8593-a5dc-4142-bd3d-5a700673f12a\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " Nov 04 02:55:22.425042 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.424938 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\") pod \"555c8593-a5dc-4142-bd3d-5a700673f12a\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " Nov 04 02:55:22.425106 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.425090 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\") pod \"555c8593-a5dc-4142-bd3d-5a700673f12a\" (UID: \"555c8593-a5dc-4142-bd3d-5a700673f12a\") " Nov 04 02:55:22.425555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.425494 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca" pod "555c8593-a5dc-4142-bd3d-5a700673f12a" (UID: "555c8593-a5dc-4142-bd3d-5a700673f12a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:22.425809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.425770 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:22.427207 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.427172 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca" pod "555c8593-a5dc-4142-bd3d-5a700673f12a" (UID: "555c8593-a5dc-4142-bd3d-5a700673f12a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:22.427300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.427277 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7" pod "555c8593-a5dc-4142-bd3d-5a700673f12a" (UID: "555c8593-a5dc-4142-bd3d-5a700673f12a"). InnerVolumeSpecName "kube-api-access-n54x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:22.528050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.528012 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54x7\" (UniqueName: \"kubernetes.io/projected/555c8593-a5dc-4142-bd3d-5a700673f12a-kube-api-access-n54x7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:22.528050 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:22.528049 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c8593-a5dc-4142-bd3d-5a700673f12a-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:23.192793 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.192759 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fedc871-3ff0-4689-a2e3-29a534cc92dd" path="/var/lib/kubelet/pods/5fedc871-3ff0-4689-a2e3-29a534cc92dd/volumes" Nov 04 02:55:23.199516 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.199490 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 02:55:23.199681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.199527 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" event={"ID":"555c8593-a5dc-4142-bd3d-5a700673f12a","Type":"ContainerDied","Data":"d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66"} Nov 04 02:55:23.199681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.199577 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d11f8c002bbf13c86a1e85c1726c9f18344e7e757086a339e6861a5008f77f66" Nov 04 02:55:23.327401 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.327357 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:23.327571 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:23.327420 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:24.327500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.327467 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:24.327919 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.327539 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:24.390512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.390469 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4"] Nov 04 02:55:24.391448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.391423 4784 cpu_manager.go:414] "containerMap: removing container" podUID="555c8593-a5dc-4142-bd3d-5a700673f12a" containerName="revoke-keys" Nov 04 02:55:24.391448 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.391445 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555c8593-a5dc-4142-bd3d-5a700673f12a" containerName="revoke-keys" Nov 04 02:55:24.391617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.391582 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="555c8593-a5dc-4142-bd3d-5a700673f12a" containerName="revoke-keys" Nov 04 02:55:24.396234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.396172 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.433329 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.433292 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4"] Nov 04 02:55:24.479493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.479460 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:55:24.480359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.479785 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:55:24.480506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.479812 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:55:24.480732 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.480715 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:55:24.484756 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.484723 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.485025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.485000 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.485253 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.485227 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dlb\" (UniqueName: \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.493117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.493088 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7027-ab164a"] Nov 04 02:55:24.498011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.497992 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:24.551902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.551863 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hzl5n\"" Nov 04 02:55:24.581347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.581248 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7027-ab164a"] Nov 04 02:55:24.588255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.588178 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.588424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.588351 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dlb\" (UniqueName: \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.588424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.588401 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7bm\" (UniqueName: \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\") pod \"test-curl-7027-ab164a\" (UID: \"5c82115c-18c7-432c-846d-175ed5357148\") " pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:24.588503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.588450 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.589107 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.589089 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.591600 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.591578 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.651864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.651828 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dlb\" (UniqueName: \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\") pod \"maas-api-revoke-sub-aaa8d90f-4ppx4\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.690593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.690555 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7bm\" (UniqueName: \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\") pod \"test-curl-7027-ab164a\" (UID: \"5c82115c-18c7-432c-846d-175ed5357148\") " pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:24.712026 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.711996 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:24.752662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.752630 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7bm\" (UniqueName: \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\") pod \"test-curl-7027-ab164a\" (UID: \"5c82115c-18c7-432c-846d-175ed5357148\") " pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:24.812757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:24.812712 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:25.141859 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:25.141818 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2961fc3_c8e4_405a_9cc2_4ebc92ece520.slice/crio-261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f WatchSource:0}: Error finding container 261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f: Status 404 returned error can't find the container with id 261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f Nov 04 02:55:25.146020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.145997 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4"] Nov 04 02:55:25.226226 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.226163 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" event={"ID":"b2961fc3-c8e4-405a-9cc2-4ebc92ece520","Type":"ContainerStarted","Data":"261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f"} Nov 04 02:55:25.242522 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:25.242492 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c82115c_18c7_432c_846d_175ed5357148.slice/crio-0539ca2ed1ccb33845d85f2b40a03e5f2a3f77e9952ae2f268eace5d03e5c689 WatchSource:0}: Error finding container 0539ca2ed1ccb33845d85f2b40a03e5f2a3f77e9952ae2f268eace5d03e5c689: Status 404 returned error can't find the container with id 0539ca2ed1ccb33845d85f2b40a03e5f2a3f77e9952ae2f268eace5d03e5c689 Nov 04 02:55:25.244404 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.244380 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7027-ab164a"] Nov 04 02:55:25.244493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.244451 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:55:25.327974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.327937 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:25.328406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:25.327991 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:26.084743 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.084639 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk"] Nov 04 02:55:26.094644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.094609 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.121414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.121372 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk"] Nov 04 02:55:26.227854 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.227823 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.228041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.227944 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.228338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.228305 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886d6\" (UniqueName: \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.237464 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.237425 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" event={"ID":"b2961fc3-c8e4-405a-9cc2-4ebc92ece520","Type":"ContainerStarted","Data":"cb044b86d65039614fee380599616823c75bf413b11dbc3d18706a8ff323d4d1"} Nov 04 02:55:26.239005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.238978 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7027-ab164a" event={"ID":"5c82115c-18c7-432c-846d-175ed5357148","Type":"ContainerStarted","Data":"0539ca2ed1ccb33845d85f2b40a03e5f2a3f77e9952ae2f268eace5d03e5c689"} Nov 04 02:55:26.327529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.327494 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:26.327817 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.327555 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:26.329440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.329416 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886d6\" (UniqueName: \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.329881 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.329505 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.329881 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.329607 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.330122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.330100 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.332102 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.332073 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.388634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.388582 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" podStartSLOduration=2.388565363 podStartE2EDuration="2.388565363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:26.354142453 +0000 UTC m=+1373.320643446" watchObservedRunningTime="2026-11-04 02:55:26.388565363 +0000 UTC m=+1373.355066354" Nov 04 02:55:26.394017 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.393987 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-886d6\" (UniqueName: \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\") pod \"maas-api-revoke-sub-7f098c46-6stkk\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.417876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.417843 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:26.479239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.479181 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft"] Nov 04 02:55:26.487474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.487446 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.518130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.518099 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft"] Nov 04 02:55:26.537300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.537274 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hv66\" (UniqueName: \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.537460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.537350 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.537617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.537586 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.638996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.638903 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hv66\" (UniqueName: \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.639171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.639047 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.639350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.639333 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.639929 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.639898 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.641862 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.641836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.701353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.701317 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hv66\" (UniqueName: \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\") pod \"maas-api-revoke-sub-aa36587c-w8sft\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.809613 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.809582 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:26.844641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:26.844607 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk"] Nov 04 02:55:26.846632 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:26.846603 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ea6d41_fe3e_459c_9f87_24c138f380c7.slice/crio-9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5 WatchSource:0}: Error finding container 9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5: Status 404 returned error can't find the container with id 9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5 Nov 04 02:55:27.253081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.253031 4784 generic.go:356] "Generic (PLEG): container finished" podID="b2961fc3-c8e4-405a-9cc2-4ebc92ece520" containerID="cb044b86d65039614fee380599616823c75bf413b11dbc3d18706a8ff323d4d1" exitCode=0 Nov 04 02:55:27.253333 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.253101 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" event={"ID":"b2961fc3-c8e4-405a-9cc2-4ebc92ece520","Type":"ContainerDied","Data":"cb044b86d65039614fee380599616823c75bf413b11dbc3d18706a8ff323d4d1"} Nov 04 02:55:27.278215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.255225 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7027-ab164a" event={"ID":"5c82115c-18c7-432c-846d-175ed5357148","Type":"ContainerStarted","Data":"724e14e1c083b851d2d331856edd0702dcd559696a606436cd84e313858a183d"} Nov 04 02:55:27.286044 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.286008 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" event={"ID":"27ea6d41-fe3e-459c-9f87-24c138f380c7","Type":"ContainerStarted","Data":"9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5"} Nov 04 02:55:27.289845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.289815 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft"] Nov 04 02:55:27.328442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.328416 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:27.328548 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.328465 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:27.501244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:27.501182 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7027-ab164a" podStartSLOduration=1.8583328479999999 podStartE2EDuration="3.501168006s" totalImagesPullingTime="1.642835158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:27.498817954 +0000 UTC m=+1374.465318946" watchObservedRunningTime="2026-11-04 02:55:27.501168006 +0000 UTC m=+1374.467668997" Nov 04 02:55:28.311301 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.311156 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" event={"ID":"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af","Type":"ContainerStarted","Data":"332d78da0c78348eda88eeee9682ba1db408f3209d2644e521131ebf29ce5dff"} Nov 04 02:55:28.311714 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.311683 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" event={"ID":"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af","Type":"ContainerStarted","Data":"4d4ae5bb6d9e6ca6093c64ce0b14e7e722321d867e572140f1b3dc72dba92ea8"} Nov 04 02:55:28.314768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.314737 4784 generic.go:356] "Generic (PLEG): container finished" podID="27ea6d41-fe3e-459c-9f87-24c138f380c7" containerID="657f40b58e60e146f8f4cded3aa9517eaedc0dc39934ece6ea241fb089796e01" exitCode=0 Nov 04 02:55:28.314932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.314867 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" event={"ID":"27ea6d41-fe3e-459c-9f87-24c138f380c7","Type":"ContainerDied","Data":"657f40b58e60e146f8f4cded3aa9517eaedc0dc39934ece6ea241fb089796e01"} Nov 04 02:55:28.327327 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.327291 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 02:55:28.327453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.327349 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 02:55:28.433778 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.433684 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" podStartSLOduration=2.43366591 podStartE2EDuration="2.43366591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:28.430603631 +0000 UTC m=+1375.397104625" watchObservedRunningTime="2026-11-04 02:55:28.43366591 +0000 UTC m=+1375.400166915" Nov 04 02:55:28.510045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.510019 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:28.602028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.601941 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\" (UniqueName: \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\") pod \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " Nov 04 02:55:28.602255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.602072 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\") pod \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " Nov 04 02:55:28.602255 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.602149 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\") pod \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\" (UID: \"b2961fc3-c8e4-405a-9cc2-4ebc92ece520\") " Nov 04 02:55:28.602538 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.602510 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca" pod "b2961fc3-c8e4-405a-9cc2-4ebc92ece520" (UID: "b2961fc3-c8e4-405a-9cc2-4ebc92ece520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:28.602870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.602850 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:28.604640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.604616 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca" pod "b2961fc3-c8e4-405a-9cc2-4ebc92ece520" (UID: "b2961fc3-c8e4-405a-9cc2-4ebc92ece520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:28.604927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.604908 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb" pod "b2961fc3-c8e4-405a-9cc2-4ebc92ece520" (UID: "b2961fc3-c8e4-405a-9cc2-4ebc92ece520"). InnerVolumeSpecName "kube-api-access-c2dlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:28.704944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.704912 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:28.704944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.704945 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2dlb\" (UniqueName: \"kubernetes.io/projected/b2961fc3-c8e4-405a-9cc2-4ebc92ece520-kube-api-access-c2dlb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:28.985244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.978041 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7027-ab164a"] Nov 04 02:55:28.988921 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:28.988883 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7027-ab164a"] Nov 04 02:55:29.130501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.130460 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7027-c6b01e"] Nov 04 02:55:29.131395 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131369 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b2961fc3-c8e4-405a-9cc2-4ebc92ece520" containerName="revoke-keys" Nov 04 02:55:29.131395 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131394 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2961fc3-c8e4-405a-9cc2-4ebc92ece520" containerName="revoke-keys" Nov 04 02:55:29.131634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131471 4784 cpu_manager.go:414] "containerMap: removing container" podUID="5c82115c-18c7-432c-846d-175ed5357148" containerName="test-curl-7027-ab164a" Nov 04 02:55:29.131634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131479 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c82115c-18c7-432c-846d-175ed5357148" containerName="test-curl-7027-ab164a" Nov 04 02:55:29.131770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131648 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c82115c-18c7-432c-846d-175ed5357148" containerName="test-curl-7027-ab164a" Nov 04 02:55:29.131770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.131669 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2961fc3-c8e4-405a-9cc2-4ebc92ece520" containerName="revoke-keys" Nov 04 02:55:29.136322 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.136297 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:29.190725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.190684 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7027-c6b01e"] Nov 04 02:55:29.313874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.313780 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxlkd\" (UniqueName: \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\") pod \"test-curl-7027-c6b01e\" (UID: \"c1955801-9783-4bc3-90e0-edf7068d7c88\") " pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:29.327582 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.327546 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" Nov 04 02:55:29.327951 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.327651 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4" event={"ID":"b2961fc3-c8e4-405a-9cc2-4ebc92ece520","Type":"ContainerDied","Data":"261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f"} Nov 04 02:55:29.327951 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.327717 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261f2d1d5966dea53fb806d0a3fc1c31857b50eb99f7e1f71c6a350f9b0f3e3f" Nov 04 02:55:29.330128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.330095 4784 generic.go:356] "Generic (PLEG): container finished" podID="cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" containerID="332d78da0c78348eda88eeee9682ba1db408f3209d2644e521131ebf29ce5dff" exitCode=0 Nov 04 02:55:29.330465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.330439 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7027-ab164a" podUID="5c82115c-18c7-432c-846d-175ed5357148" containerName="test-curl-7027-ab164a" containerID="cri-o://724e14e1c083b851d2d331856edd0702dcd559696a606436cd84e313858a183d" gracePeriod=2 Nov 04 02:55:29.330842 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.330817 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" event={"ID":"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af","Type":"ContainerDied","Data":"332d78da0c78348eda88eeee9682ba1db408f3209d2644e521131ebf29ce5dff"} Nov 04 02:55:29.380697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.380657 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k"] Nov 04 02:55:29.386804 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.386772 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.410339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.410314 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:29.410640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.410620 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:29.411579 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.411552 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 02:55:29.415388 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.415361 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxlkd\" (UniqueName: \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\") pod \"test-curl-7027-c6b01e\" (UID: \"c1955801-9783-4bc3-90e0-edf7068d7c88\") " pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:29.466020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.465991 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k"] Nov 04 02:55:29.481874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.481842 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxlkd\" (UniqueName: \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\") pod \"test-curl-7027-c6b01e\" (UID: \"c1955801-9783-4bc3-90e0-edf7068d7c88\") " pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:29.517244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.517211 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdsg\" (UniqueName: \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.517726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.517278 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.517726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.517690 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.556413 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.556383 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:29.619936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.619850 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.620122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.620008 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdsg\" (UniqueName: \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.620122 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.620038 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.620666 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.620641 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.623293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.623265 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.682296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.682268 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdsg\" (UniqueName: \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\") pod \"maas-api-revoke-sub-355cbc8a-wsv5k\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.705782 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.705747 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:29.721891 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.721852 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\") pod \"27ea6d41-fe3e-459c-9f87-24c138f380c7\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " Nov 04 02:55:29.722171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.722148 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\" (UniqueName: \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\") pod \"27ea6d41-fe3e-459c-9f87-24c138f380c7\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " Nov 04 02:55:29.722272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.722223 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\") pod \"27ea6d41-fe3e-459c-9f87-24c138f380c7\" (UID: \"27ea6d41-fe3e-459c-9f87-24c138f380c7\") " Nov 04 02:55:29.732468 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.725134 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca" pod "27ea6d41-fe3e-459c-9f87-24c138f380c7" (UID: "27ea6d41-fe3e-459c-9f87-24c138f380c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:29.739797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.739743 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6" pod "27ea6d41-fe3e-459c-9f87-24c138f380c7" (UID: "27ea6d41-fe3e-459c-9f87-24c138f380c7"). InnerVolumeSpecName "kube-api-access-886d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:29.742447 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.742415 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca" pod "27ea6d41-fe3e-459c-9f87-24c138f380c7" (UID: "27ea6d41-fe3e-459c-9f87-24c138f380c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:29.753735 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.753706 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:29.825798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.825550 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886d6\" (UniqueName: \"kubernetes.io/projected/27ea6d41-fe3e-459c-9f87-24c138f380c7-kube-api-access-886d6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:29.825798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.825582 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:29.825798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:29.825593 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ea6d41-fe3e-459c-9f87-24c138f380c7-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:30.134795 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:30.134761 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830882a3_067d_49bf_ade4_d486a6b166eb.slice/crio-5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab WatchSource:0}: Error finding container 5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab: Status 404 returned error can't find the container with id 5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab Nov 04 02:55:30.145380 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.145356 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k"] Nov 04 02:55:30.183929 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:30.183902 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1955801_9783_4bc3_90e0_edf7068d7c88.slice/crio-20adb19090285357e5bd58de60c0f318ffb6ccf9e6ffd8d49684183940008b67 WatchSource:0}: Error finding container 20adb19090285357e5bd58de60c0f318ffb6ccf9e6ffd8d49684183940008b67: Status 404 returned error can't find the container with id 20adb19090285357e5bd58de60c0f318ffb6ccf9e6ffd8d49684183940008b67 Nov 04 02:55:30.186171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.186149 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7027-c6b01e"] Nov 04 02:55:30.343103 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.343056 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7027-c6b01e" event={"ID":"c1955801-9783-4bc3-90e0-edf7068d7c88","Type":"ContainerStarted","Data":"877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072"} Nov 04 02:55:30.343339 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.343118 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7027-c6b01e" event={"ID":"c1955801-9783-4bc3-90e0-edf7068d7c88","Type":"ContainerStarted","Data":"20adb19090285357e5bd58de60c0f318ffb6ccf9e6ffd8d49684183940008b67"} Nov 04 02:55:30.345626 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.345598 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" event={"ID":"830882a3-067d-49bf-ade4-d486a6b166eb","Type":"ContainerStarted","Data":"5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab"} Nov 04 02:55:30.347111 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.347088 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" Nov 04 02:55:30.347252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.347094 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk" event={"ID":"27ea6d41-fe3e-459c-9f87-24c138f380c7","Type":"ContainerDied","Data":"9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5"} Nov 04 02:55:30.347252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.347246 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9528481c49cedf990696df95146c67bb18d2dc169c59dfe0467c9eee30fe26c5" Nov 04 02:55:30.499252 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.499227 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:30.528052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.527991 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7027-c6b01e" podStartSLOduration=1.527972176 podStartE2EDuration="1.527972176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:30.525894497 +0000 UTC m=+1377.492395488" watchObservedRunningTime="2026-11-04 02:55:30.527972176 +0000 UTC m=+1377.494473171" Nov 04 02:55:30.642153 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.642115 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\") pod \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " Nov 04 02:55:30.642335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.642301 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\") pod \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " Nov 04 02:55:30.642511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.642339 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\" (UniqueName: \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\") pod \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\" (UID: \"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af\") " Nov 04 02:55:30.642511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.642487 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca" pod "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" (UID: "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:30.648024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.647987 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:30.651289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.651241 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66" pod "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" (UID: "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af"). InnerVolumeSpecName "kube-api-access-8hv66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:30.661490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.660097 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca" pod "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" (UID: "cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:30.749728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.749690 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:30.749728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:30.749731 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hv66\" (UniqueName: \"kubernetes.io/projected/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af-kube-api-access-8hv66\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:31.365059 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.365022 4784 generic.go:356] "Generic (PLEG): container finished" podID="5c82115c-18c7-432c-846d-175ed5357148" containerID="724e14e1c083b851d2d331856edd0702dcd559696a606436cd84e313858a183d" exitCode=137 Nov 04 02:55:31.367036 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.367010 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" Nov 04 02:55:31.367161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.367007 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft" event={"ID":"cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af","Type":"ContainerDied","Data":"4d4ae5bb6d9e6ca6093c64ce0b14e7e722321d867e572140f1b3dc72dba92ea8"} Nov 04 02:55:31.367161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.367142 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4ae5bb6d9e6ca6093c64ce0b14e7e722321d867e572140f1b3dc72dba92ea8" Nov 04 02:55:31.369025 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.368998 4784 generic.go:356] "Generic (PLEG): container finished" podID="830882a3-067d-49bf-ade4-d486a6b166eb" containerID="9a69d3265de8f688b0142cb134fe040bdcd3d200bc10cf7e3abdafddcda4c25e" exitCode=0 Nov 04 02:55:31.369131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.369075 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" event={"ID":"830882a3-067d-49bf-ade4-d486a6b166eb","Type":"ContainerDied","Data":"9a69d3265de8f688b0142cb134fe040bdcd3d200bc10cf7e3abdafddcda4c25e"} Nov 04 02:55:31.582830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.582803 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:31.770845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.770759 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\" (UniqueName: \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\") pod \"5c82115c-18c7-432c-846d-175ed5357148\" (UID: \"5c82115c-18c7-432c-846d-175ed5357148\") " Nov 04 02:55:31.773167 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.773137 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm" pod "5c82115c-18c7-432c-846d-175ed5357148" (UID: "5c82115c-18c7-432c-846d-175ed5357148"). InnerVolumeSpecName "kube-api-access-th7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:31.872841 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:31.872801 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7bm\" (UniqueName: \"kubernetes.io/projected/5c82115c-18c7-432c-846d-175ed5357148-kube-api-access-th7bm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:32.057273 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.055878 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7027-c6b01e"] Nov 04 02:55:32.080521 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.080482 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7027-c6b01e"] Nov 04 02:55:32.381086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.381000 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:32.381277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.381088 4784 scope.go:122] "RemoveContainer" containerID="724e14e1c083b851d2d331856edd0702dcd559696a606436cd84e313858a183d" Nov 04 02:55:32.381943 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.381480 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7027-c6b01e" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" containerName="test-curl-7027-c6b01e" containerID="cri-o://877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072" gracePeriod=2 Nov 04 02:55:32.434156 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:32.434120 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-c6b01e\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:32.486667 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:32.486623 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-ab164a\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="5c82115c-18c7-432c-846d-175ed5357148" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:32.538993 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:32.538962 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-c6b01e\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:32.560551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.560522 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:32.583894 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.583861 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\") pod \"830882a3-067d-49bf-ade4-d486a6b166eb\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " Nov 04 02:55:32.584311 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.583918 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\" (UniqueName: \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\") pod \"830882a3-067d-49bf-ade4-d486a6b166eb\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " Nov 04 02:55:32.584311 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.584075 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\") pod \"830882a3-067d-49bf-ade4-d486a6b166eb\" (UID: \"830882a3-067d-49bf-ade4-d486a6b166eb\") " Nov 04 02:55:32.584396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.584303 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca" pod "830882a3-067d-49bf-ade4-d486a6b166eb" (UID: "830882a3-067d-49bf-ade4-d486a6b166eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:32.584770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.584750 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:32.586407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.586273 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca" pod "830882a3-067d-49bf-ade4-d486a6b166eb" (UID: "830882a3-067d-49bf-ade4-d486a6b166eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:32.586407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.586388 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg" pod "830882a3-067d-49bf-ade4-d486a6b166eb" (UID: "830882a3-067d-49bf-ade4-d486a6b166eb"). InnerVolumeSpecName "kube-api-access-9tdsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:32.613226 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:32.613171 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-c6b01e\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:32.686047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.686015 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tdsg\" (UniqueName: \"kubernetes.io/projected/830882a3-067d-49bf-ade4-d486a6b166eb-kube-api-access-9tdsg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:32.686047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:32.686045 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830882a3-067d-49bf-ade4-d486a6b166eb-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:32.782710 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:32.782667 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-ab164a\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="5c82115c-18c7-432c-846d-175ed5357148" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:33.171961 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.171923 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c82115c-18c7-432c-846d-175ed5357148" path="/var/lib/kubelet/pods/5c82115c-18c7-432c-846d-175ed5357148/volumes" Nov 04 02:55:33.213952 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:33.213908 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-ab164a\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="5c82115c-18c7-432c-846d-175ed5357148" pod="opendatahub/test-curl-7027-ab164a" Nov 04 02:55:33.266252 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:33.266216 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7027-c6b01e\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:33.394100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.394070 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" Nov 04 02:55:33.394100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.394084 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k" event={"ID":"830882a3-067d-49bf-ade4-d486a6b166eb","Type":"ContainerDied","Data":"5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab"} Nov 04 02:55:33.394326 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.394146 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f89bfd1ce2dbd995730724c19e0a9a6e34592e8046750d5435befccbcf2dbab" Nov 04 02:55:33.455581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.455490 4784 scope.go:122] "RemoveContainer" containerID="8d5a3ad0fa950e6dbc791ddc96b1ae4fe0db1f29eaae31d86b332d8116104033" Nov 04 02:55:33.467440 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.467417 4784 scope.go:122] "RemoveContainer" containerID="cbc3615fa1394be5eb0e17209bebe6cbb98f82044d18f4f49d0c16f69f66737a" Nov 04 02:55:33.479046 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.479026 4784 scope.go:122] "RemoveContainer" containerID="dd65eac7ba1b965aa9edf5792292834c502afb06cd431abd894ef708466ba4ec" Nov 04 02:55:33.489915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.489886 4784 scope.go:122] "RemoveContainer" containerID="8752bca1cd6f8e518812950adfeb7b55b58a0c84a86e6e7009de56757dae21b3" Nov 04 02:55:33.501178 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.501157 4784 scope.go:122] "RemoveContainer" containerID="2b0818927bb8e8c1d9e7cb01611492a47e426b0e01feecea10986e940adef9c6" Nov 04 02:55:33.512293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.512270 4784 scope.go:122] "RemoveContainer" containerID="a98224efe92316b9772e60677d57204dcdf80f2e21ca763d72a11b07d6ce4b46" Nov 04 02:55:33.523318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.523289 4784 scope.go:122] "RemoveContainer" containerID="cb4c1add07fd0945645ab540e4595513da61ecb45e28735bf8edadb6d93e2cd9" Nov 04 02:55:33.707793 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.707699 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:55:33.708457 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708441 4784 cpu_manager.go:414] "containerMap: removing container" podUID="cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" containerName="revoke-keys" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708458 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" containerName="revoke-keys" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708477 4784 cpu_manager.go:414] "containerMap: removing container" podUID="830882a3-067d-49bf-ade4-d486a6b166eb" containerName="revoke-keys" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708482 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830882a3-067d-49bf-ade4-d486a6b166eb" containerName="revoke-keys" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708505 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" containerName="test-curl-7027-c6b01e" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708511 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" containerName="test-curl-7027-c6b01e" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708521 4784 cpu_manager.go:414] "containerMap: removing container" podUID="27ea6d41-fe3e-459c-9f87-24c138f380c7" containerName="revoke-keys" Nov 04 02:55:33.708546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708525 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ea6d41-fe3e-459c-9f87-24c138f380c7" containerName="revoke-keys" Nov 04 02:55:33.708761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708639 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" containerName="revoke-keys" Nov 04 02:55:33.708761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708649 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="830882a3-067d-49bf-ade4-d486a6b166eb" containerName="revoke-keys" Nov 04 02:55:33.708761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708659 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" containerName="test-curl-7027-c6b01e" Nov 04 02:55:33.708761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.708665 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ea6d41-fe3e-459c-9f87-24c138f380c7" containerName="revoke-keys" Nov 04 02:55:33.714935 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.714908 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.805805 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.805760 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.805822 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.805867 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806005 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.805947 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.806013 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.806056 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.806244 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.806280 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.806462 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.806385 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8dv\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.837450 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.836616 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:55:33.879215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.870037 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2c1c4f09-gw-openshift-default-dockercfg-lnfmt\"" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.907803 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.907874 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.907978 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908058 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908103 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908243 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908277 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908285 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908381 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8dv\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908438 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908585 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.908890 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.911781 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.909597 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.916129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.911814 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.916129 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.912181 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:33.987302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:33.984053 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:34.005443 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.004365 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8dv\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\") pod \"e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:34.057845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.057818 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:34.531541 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:34.531512 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dec8ab1_099a_4e1b_b525_2a430f676947.slice/crio-cd686dee959bdef23209a8c30e52aa166755a06e25a743487831d9df4619c372 WatchSource:0}: Error finding container cd686dee959bdef23209a8c30e52aa166755a06e25a743487831d9df4619c372: Status 404 returned error can't find the container with id cd686dee959bdef23209a8c30e52aa166755a06e25a743487831d9df4619c372 Nov 04 02:55:34.534489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.534466 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:55:34.534617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.534586 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:34.534669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.534657 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:34.534701 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.534685 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:34.633623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.633569 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:34.724792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.724756 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\" (UniqueName: \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\") pod \"c1955801-9783-4bc3-90e0-edf7068d7c88\" (UID: \"c1955801-9783-4bc3-90e0-edf7068d7c88\") " Nov 04 02:55:34.727390 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.727359 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd" pod "c1955801-9783-4bc3-90e0-edf7068d7c88" (UID: "c1955801-9783-4bc3-90e0-edf7068d7c88"). InnerVolumeSpecName "kube-api-access-nxlkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:34.827793 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:34.827749 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxlkd\" (UniqueName: \"kubernetes.io/projected/c1955801-9783-4bc3-90e0-edf7068d7c88-kube-api-access-nxlkd\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:35.165946 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.165911 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1955801-9783-4bc3-90e0-edf7068d7c88" path="/var/lib/kubelet/pods/c1955801-9783-4bc3-90e0-edf7068d7c88/volumes" Nov 04 02:55:35.444681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.444593 4784 generic.go:356] "Generic (PLEG): container finished" podID="c1955801-9783-4bc3-90e0-edf7068d7c88" containerID="877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072" exitCode=137 Nov 04 02:55:35.444681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.444657 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7027-c6b01e" Nov 04 02:55:35.444904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.444677 4784 scope.go:122] "RemoveContainer" containerID="877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072" Nov 04 02:55:35.447154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.447128 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerStarted","Data":"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253"} Nov 04 02:55:35.447348 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.447236 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerStarted","Data":"cd686dee959bdef23209a8c30e52aa166755a06e25a743487831d9df4619c372"} Nov 04 02:55:35.457539 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.457522 4784 scope.go:122] "RemoveContainer" containerID="877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072" Nov 04 02:55:35.457837 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:35.457813 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072\": container with ID starting with 877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072 not found: ID does not exist" containerID="877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072" Nov 04 02:55:35.457936 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.457847 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072"} err="failed to get container status \"877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072\": rpc error: code = NotFound desc = could not find container \"877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072\": container with ID starting with 877915ef5d79936373aa31c09fd5f31a6ab61435b1a34ed1085d5b594e078072 not found: ID does not exist" Nov 04 02:55:35.573330 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:35.573267 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podStartSLOduration=2.573248884 podStartE2EDuration="2.573248884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:35.571514026 +0000 UTC m=+1382.538015018" watchObservedRunningTime="2026-11-04 02:55:35.573248884 +0000 UTC m=+1382.539749879" Nov 04 02:55:36.058646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:36.058615 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:36.064108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:36.064080 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:36.439211 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:36.432668 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:55:36.499324 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:36.499270 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:36.508395 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:36.506595 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:55:37.332942 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.332904 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl"] Nov 04 02:55:37.338924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.338901 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.389634 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.389593 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.389864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.389679 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjt5z\" (UniqueName: \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.389864 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.389730 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.392072 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.392022 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:55:37.392509 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.392481 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:55:37.392641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.392591 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:55:37.405593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.405558 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:55:37.444248 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.443850 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl"] Nov 04 02:55:37.492108 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.492067 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.492362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.492144 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjt5z\" (UniqueName: \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.492362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.492208 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.493098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.493069 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.517584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.516414 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.578651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.578236 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjt5z\" (UniqueName: \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\") pod \"maas-api-revoke-sub-ef7fd5cb-4zvfl\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:37.654819 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:37.654732 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:38.064468 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:38.064437 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535e4550_706e_4dea_bac1_f057ae31abfa.slice/crio-9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca WatchSource:0}: Error finding container 9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca: Status 404 returned error can't find the container with id 9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca Nov 04 02:55:38.065254 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:38.065230 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl"] Nov 04 02:55:38.512043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:38.512008 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" event={"ID":"535e4550-706e-4dea-bac1-f057ae31abfa","Type":"ContainerStarted","Data":"3727f6823c1711bb0bf2628ec74809fca5bc79421bcc0a9c209e90e29591a6b7"} Nov 04 02:55:38.512573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:38.512064 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" event={"ID":"535e4550-706e-4dea-bac1-f057ae31abfa","Type":"ContainerStarted","Data":"9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca"} Nov 04 02:55:38.634776 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:38.634674 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" podStartSLOduration=1.6346603929999999 podStartE2EDuration="1.634660393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:38.631927181 +0000 UTC m=+1385.598428167" watchObservedRunningTime="2026-11-04 02:55:38.634660393 +0000 UTC m=+1385.601161384" Nov 04 02:55:39.527980 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:39.527929 4784 generic.go:356] "Generic (PLEG): container finished" podID="535e4550-706e-4dea-bac1-f057ae31abfa" containerID="3727f6823c1711bb0bf2628ec74809fca5bc79421bcc0a9c209e90e29591a6b7" exitCode=0 Nov 04 02:55:39.528525 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:39.528033 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" event={"ID":"535e4550-706e-4dea-bac1-f057ae31abfa","Type":"ContainerDied","Data":"3727f6823c1711bb0bf2628ec74809fca5bc79421bcc0a9c209e90e29591a6b7"} Nov 04 02:55:40.407547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.407514 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5"] Nov 04 02:55:40.412335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.412304 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.469941 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.469908 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5"] Nov 04 02:55:40.571232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.570671 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.571616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.571279 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.571616 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.571364 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4l7h\" (UniqueName: \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.672736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.672691 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4l7h\" (UniqueName: \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.673043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.673011 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.673150 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.673072 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.673884 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.673858 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.675623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.675605 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.716816 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.716791 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:40.735387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.735357 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4l7h\" (UniqueName: \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\") pod \"maas-api-revoke-sub-33af2588-5xsk5\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:40.875624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.875589 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\") pod \"535e4550-706e-4dea-bac1-f057ae31abfa\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " Nov 04 02:55:40.875808 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.875639 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\" (UniqueName: \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\") pod \"535e4550-706e-4dea-bac1-f057ae31abfa\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " Nov 04 02:55:40.875852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.875823 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\") pod \"535e4550-706e-4dea-bac1-f057ae31abfa\" (UID: \"535e4550-706e-4dea-bac1-f057ae31abfa\") " Nov 04 02:55:40.876232 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.876161 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca" pod "535e4550-706e-4dea-bac1-f057ae31abfa" (UID: "535e4550-706e-4dea-bac1-f057ae31abfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:40.876668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.876646 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:40.877982 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.877957 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca" pod "535e4550-706e-4dea-bac1-f057ae31abfa" (UID: "535e4550-706e-4dea-bac1-f057ae31abfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:40.878159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.878133 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z" pod "535e4550-706e-4dea-bac1-f057ae31abfa" (UID: "535e4550-706e-4dea-bac1-f057ae31abfa"). InnerVolumeSpecName "kube-api-access-jjt5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:40.977820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.977725 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535e4550-706e-4dea-bac1-f057ae31abfa-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:40.977820 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:40.977774 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjt5z\" (UniqueName: \"kubernetes.io/projected/535e4550-706e-4dea-bac1-f057ae31abfa-kube-api-access-jjt5z\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:41.028852 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.028820 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:41.414844 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:41.414812 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c55449_0261_41f0_b602_ac86f3706e01.slice/crio-754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85 WatchSource:0}: Error finding container 754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85: Status 404 returned error can't find the container with id 754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85 Nov 04 02:55:41.417099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.417076 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5"] Nov 04 02:55:41.550998 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.550961 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" event={"ID":"74c55449-0261-41f0-b602-ac86f3706e01","Type":"ContainerStarted","Data":"754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85"} Nov 04 02:55:41.554051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.552993 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" Nov 04 02:55:41.554051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.553016 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl" event={"ID":"535e4550-706e-4dea-bac1-f057ae31abfa","Type":"ContainerDied","Data":"9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca"} Nov 04 02:55:41.554051 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:41.553062 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9119a3fcea174dc8b8fff70cb5e0682cf4d7c312eac4696c95011f858236a7ca" Nov 04 02:55:42.584220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:42.578961 4784 generic.go:356] "Generic (PLEG): container finished" podID="74c55449-0261-41f0-b602-ac86f3706e01" containerID="099e1f4cef6285bf4b1170acaf531e2ae00d7bb865b4823a1efe792aceabb83f" exitCode=0 Nov 04 02:55:42.584220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:42.579165 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" event={"ID":"74c55449-0261-41f0-b602-ac86f3706e01","Type":"ContainerDied","Data":"099e1f4cef6285bf4b1170acaf531e2ae00d7bb865b4823a1efe792aceabb83f"} Nov 04 02:55:44.572214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:44.566346 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:44.959214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:44.951375 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.016932 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.016885 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.168416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.168371 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:45.172214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.054304 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.184749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.142940 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:45.184749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.143274 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.184749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.143777 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:45.184749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.033159 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.209493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.208627 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:45.576555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.567413 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.576555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.568633 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:55:45.775551 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.775520 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:45.867971 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.867869 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\") pod \"74c55449-0261-41f0-b602-ac86f3706e01\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " Nov 04 02:55:45.868839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.868814 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\") pod \"74c55449-0261-41f0-b602-ac86f3706e01\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " Nov 04 02:55:45.895597 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.895568 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\" (UniqueName: \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\") pod \"74c55449-0261-41f0-b602-ac86f3706e01\" (UID: \"74c55449-0261-41f0-b602-ac86f3706e01\") " Nov 04 02:55:45.897004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.868585 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca" pod "74c55449-0261-41f0-b602-ac86f3706e01" (UID: "74c55449-0261-41f0-b602-ac86f3706e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:45.961410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.952282 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.961410 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:45.952640 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:55:46.173285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.053634 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:46.173285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.168627 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:46.173623 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.173587 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.175659 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.175626 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:46.176928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.176906 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:55:46.177117 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.177103 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:55:46.185342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.185307 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:55:46.192489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.192449 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.194145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.194110 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 02:55:46.195428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.195407 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:55:46.195628 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.195613 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:55:46.222215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.214540 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.273327 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.216712 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.216759 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.249379 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h" pod "74c55449-0261-41f0-b602-ac86f3706e01" (UID: "74c55449-0261-41f0-b602-ac86f3706e01"). InnerVolumeSpecName "kube-api-access-h4l7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.249485 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca" pod "74c55449-0261-41f0-b602-ac86f3706e01" (UID: "74c55449-0261-41f0-b602-ac86f3706e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.279620 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:55:46.287275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.279680 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:55:46.409218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.405416 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c55449-0261-41f0-b602-ac86f3706e01-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:46.409218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.405450 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4l7h\" (UniqueName: \"kubernetes.io/projected/74c55449-0261-41f0-b602-ac86f3706e01-kube-api-access-h4l7h\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:46.597534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.597432 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.783145 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.783118 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" Nov 04 02:55:46.808384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.783503 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5" event={"ID":"74c55449-0261-41f0-b602-ac86f3706e01","Type":"ContainerDied","Data":"754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85"} Nov 04 02:55:46.808954 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.808750 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754aeda9f4b9b8b6d18859abe3ccde9af3e31196eb900a043e934121c3cd0e85" Nov 04 02:55:46.967412 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:46.967354 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.247278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.222650 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.247278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.222994 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:47.247278 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.223208 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:55:47.249759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.249715 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:47.354564 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.264271 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:47.589112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.269731 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:47.589112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.287796 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:47.589112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.287829 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:47.589112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.387087 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:47.589112 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.570180 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.769743 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:47.769696 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.100258 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.094748 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:55:48.178140 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.178096 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:48.178465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.178432 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.204350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.204301 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.518218 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.486231 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.000687 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:55:48.560511 4784 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.394s" Nov 04 02:55:49.021085 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.764769 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.764985 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.765520 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/health\": context deadline exceeded" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.948679 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.948736 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.948788 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.948970 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:49.082293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:48.949170 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.106309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:49.106264 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 04 02:55:49.108377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:49.108343 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:55:49.276631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:49.276478 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.576275 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:49.102693 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.969009 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:49.968907 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.174389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043415 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043452 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043474 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043494 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043512 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043531 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.043550 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.075037 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187671 4784 cpu_manager.go:414] "containerMap: removing container" podUID="535e4550-706e-4dea-bac1-f057ae31abfa" containerName="revoke-keys" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187693 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535e4550-706e-4dea-bac1-f057ae31abfa" containerName="revoke-keys" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187725 4784 cpu_manager.go:414] "containerMap: removing container" podUID="74c55449-0261-41f0-b602-ac86f3706e01" containerName="revoke-keys" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187732 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c55449-0261-41f0-b602-ac86f3706e01" containerName="revoke-keys" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187939 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c55449-0261-41f0-b602-ac86f3706e01" containerName="revoke-keys" Nov 04 02:55:50.219280 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.187968 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="535e4550-706e-4dea-bac1-f057ae31abfa" containerName="revoke-keys" Nov 04 02:55:50.239979 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.132685 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:55:50.427552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.267549 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:50.427552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.362923 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2c1c4f09\"" Nov 04 02:55:50.427552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.363046 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2c1c4f09-dockercfg-4c8rd\"" Nov 04 02:55:50.556269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.550113 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:55:50.556269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.550704 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:50.556269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.550808 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226hw\" (UniqueName: \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:50.600109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.600064 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 02:55:50.768522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.765076 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:50.768522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.765174 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226hw\" (UniqueName: \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:50.829506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.788463 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded" Nov 04 02:55:50.899332 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:50.880383 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 04 02:55:51.187240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.178816 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.187240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.179104 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:55:51.230449 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.230399 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.239272 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.231808 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:55:51.406844 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.239585 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:55:51.407350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.239624 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:55:51.407724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.239642 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:55:51.435973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.272881 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.435973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.272965 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:55:51.435973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.282043 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:51.435973 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.364251 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226hw\" (UniqueName: \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\") pod \"payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:51.467277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.457969 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:51.600512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.600459 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.731376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.731288 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:55:51.732310 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.732117 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:55:51.733302 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.733279 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:55:51.733911 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.733889 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:55:51.734541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.734499 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:55:51.847465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.847431 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.946697 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.946736 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.946967 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.947176 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.947417 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:55:51.949028 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.947570 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:55:51.952426 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.951242 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:51.988342 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:51.988260 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz"] Nov 04 02:55:52.000809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.000775 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz"] Nov 04 02:55:52.000968 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.000898 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.054469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.054420 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:55:52.054469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.054443 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:55:52.054737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.054511 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:55:52.054737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.054571 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:55:52.141780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.141746 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmddc\" (UniqueName: \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.142001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.141976 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.142075 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.142035 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.142130 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.142075 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.142239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.142221 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t6x\" (UniqueName: \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.243918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.243831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.243918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.243880 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.243918 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.243901 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.244644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.243954 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t6x\" (UniqueName: \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.244644 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.244276 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmddc\" (UniqueName: \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.244762 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.244688 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.244845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.244820 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.247489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.247472 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.306398 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.306366 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t6x\" (UniqueName: \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\") pod \"maas-api-revoke-sub-77a76cd7-whsrz\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.306559 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.306458 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmddc\" (UniqueName: \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\") pod \"payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.327015 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.326981 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.336874 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.336847 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:52.375789 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.375707 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:55:52.385892 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:52.385858 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135614f9_124c_41c2_b7c5_b579e82ae0d5.slice/crio-e1c8e7780bd7c43d57664bd9cc4cfeab48432d87fd99b1106d837e52b50e620e WatchSource:0}: Error finding container e1c8e7780bd7c43d57664bd9cc4cfeab48432d87fd99b1106d837e52b50e620e: Status 404 returned error can't find the container with id e1c8e7780bd7c43d57664bd9cc4cfeab48432d87fd99b1106d837e52b50e620e Nov 04 02:55:52.745508 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:52.745473 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bc2e69_a1ec_403c_af02_79053de37dd2.slice/crio-985d6bd8983ef99afa95d56666f510cab7e59096cc75d2d88e6d2646f24e0eb8 WatchSource:0}: Error finding container 985d6bd8983ef99afa95d56666f510cab7e59096cc75d2d88e6d2646f24e0eb8: Status 404 returned error can't find the container with id 985d6bd8983ef99afa95d56666f510cab7e59096cc75d2d88e6d2646f24e0eb8 Nov 04 02:55:52.751351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.751286 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:55:52.804703 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:52.804666 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a42e57_9048_4e3d_a6b8_c23343d5236a.slice/crio-dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3 WatchSource:0}: Error finding container dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3: Status 404 returned error can't find the container with id dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3 Nov 04 02:55:52.806689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.806660 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz"] Nov 04 02:55:52.978493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.978445 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" event={"ID":"48a42e57-9048-4e3d-a6b8-c23343d5236a","Type":"ContainerStarted","Data":"dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3"} Nov 04 02:55:52.980809 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.980781 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" event={"ID":"02bc2e69-a1ec-403c-af02-79053de37dd2","Type":"ContainerStarted","Data":"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13"} Nov 04 02:55:52.980968 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.980832 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" event={"ID":"02bc2e69-a1ec-403c-af02-79053de37dd2","Type":"ContainerStarted","Data":"985d6bd8983ef99afa95d56666f510cab7e59096cc75d2d88e6d2646f24e0eb8"} Nov 04 02:55:52.980968 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.980862 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:55:52.982706 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.982665 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" event={"ID":"135614f9-124c-41c2-b7c5-b579e82ae0d5","Type":"ContainerStarted","Data":"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667"} Nov 04 02:55:52.982825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.982724 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" event={"ID":"135614f9-124c-41c2-b7c5-b579e82ae0d5","Type":"ContainerStarted","Data":"e1c8e7780bd7c43d57664bd9cc4cfeab48432d87fd99b1106d837e52b50e620e"} Nov 04 02:55:52.982825 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:52.982763 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:55:53.489501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:53.489446 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" podStartSLOduration=4.489429565 podStartE2EDuration="4.489429565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:53.483944716 +0000 UTC m=+1400.450445709" watchObservedRunningTime="2026-11-04 02:55:53.489429565 +0000 UTC m=+1400.455930565" Nov 04 02:55:53.630045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:53.629971 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" podStartSLOduration=5.629947137 podStartE2EDuration="5.629947137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:53.601764025 +0000 UTC m=+1400.568265019" watchObservedRunningTime="2026-11-04 02:55:53.629947137 +0000 UTC m=+1400.596448131" Nov 04 02:55:53.994393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:53.994356 4784 generic.go:356] "Generic (PLEG): container finished" podID="48a42e57-9048-4e3d-a6b8-c23343d5236a" containerID="79444fe222f1ce7f6df037a2205f231337680c26064ba479e62e161941aeef9d" exitCode=0 Nov 04 02:55:53.994590 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:53.994420 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" event={"ID":"48a42e57-9048-4e3d-a6b8-c23343d5236a","Type":"ContainerDied","Data":"79444fe222f1ce7f6df037a2205f231337680c26064ba479e62e161941aeef9d"} Nov 04 02:55:54.187977 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.187945 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 02:55:54.193158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.193132 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.247572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.247416 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 02:55:54.247572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.247460 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 02:55:54.247572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.247532 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n69rv\"" Nov 04 02:55:54.248086 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.248055 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 02:55:54.254574 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.254544 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 02:55:54.296630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.296517 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hft8t\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.296830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.296639 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.296830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.296725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.296913 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.296861 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.297029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.296999 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.297162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.297066 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.297162 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.297148 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399300 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hft8t\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399397 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399529 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399679 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399597 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399685 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399734 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.399787 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.399777 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.400770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.400744 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.402554 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.402527 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.402692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.402669 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.403329 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.402709 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.403329 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.402820 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.461996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.461914 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.462128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.462057 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hft8t\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t6h4m\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.508720 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.508686 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:55:54.949772 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:54.949741 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80054ecc_72b0_4405_8f2c_2c8c283bc3f3.slice/crio-4c19831d480139dfbc85e2f8948b3578d99628fbb6fb84abe09210fa24657be2 WatchSource:0}: Error finding container 4c19831d480139dfbc85e2f8948b3578d99628fbb6fb84abe09210fa24657be2: Status 404 returned error can't find the container with id 4c19831d480139dfbc85e2f8948b3578d99628fbb6fb84abe09210fa24657be2 Nov 04 02:55:54.952908 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:54.952880 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 02:55:55.007513 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.007473 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" event={"ID":"80054ecc-72b0-4405-8f2c-2c8c283bc3f3","Type":"ContainerStarted","Data":"4c19831d480139dfbc85e2f8948b3578d99628fbb6fb84abe09210fa24657be2"} Nov 04 02:55:55.182109 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.182086 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:55.213180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.213089 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\") pod \"48a42e57-9048-4e3d-a6b8-c23343d5236a\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " Nov 04 02:55:55.213419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.213243 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\" (UniqueName: \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\") pod \"48a42e57-9048-4e3d-a6b8-c23343d5236a\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " Nov 04 02:55:55.213419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.213278 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\") pod \"48a42e57-9048-4e3d-a6b8-c23343d5236a\" (UID: \"48a42e57-9048-4e3d-a6b8-c23343d5236a\") " Nov 04 02:55:55.213530 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.213452 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca" pod "48a42e57-9048-4e3d-a6b8-c23343d5236a" (UID: "48a42e57-9048-4e3d-a6b8-c23343d5236a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:55:55.213902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.213881 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:55.215508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.215484 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca" pod "48a42e57-9048-4e3d-a6b8-c23343d5236a" (UID: "48a42e57-9048-4e3d-a6b8-c23343d5236a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:55:55.215592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.215578 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x" pod "48a42e57-9048-4e3d-a6b8-c23343d5236a" (UID: "48a42e57-9048-4e3d-a6b8-c23343d5236a"). InnerVolumeSpecName "kube-api-access-n4t6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:55:55.319698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.319666 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4t6x\" (UniqueName: \"kubernetes.io/projected/48a42e57-9048-4e3d-a6b8-c23343d5236a-kube-api-access-n4t6x\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:55.319698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:55.319695 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a42e57-9048-4e3d-a6b8-c23343d5236a-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:55:56.020099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:56.020055 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" event={"ID":"48a42e57-9048-4e3d-a6b8-c23343d5236a","Type":"ContainerDied","Data":"dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3"} Nov 04 02:55:56.020655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:56.020163 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dffb5ec15cde38cbc345853d61bea33e37a780470d8428b2c9895a8ff3ff9fc3" Nov 04 02:55:56.020655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:56.020312 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz" Nov 04 02:55:57.249057 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.249018 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw"] Nov 04 02:55:57.249923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.249880 4784 cpu_manager.go:414] "containerMap: removing container" podUID="48a42e57-9048-4e3d-a6b8-c23343d5236a" containerName="revoke-keys" Nov 04 02:55:57.249923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.249899 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a42e57-9048-4e3d-a6b8-c23343d5236a" containerName="revoke-keys" Nov 04 02:55:57.250131 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.250034 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a42e57-9048-4e3d-a6b8-c23343d5236a" containerName="revoke-keys" Nov 04 02:55:57.254675 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.254653 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.316213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.314350 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw"] Nov 04 02:55:57.377243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.374605 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:55:57.377243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.374683 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:55:57.377243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.375387 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:55:57.377243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.375483 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:55:57.392405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.385652 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.392405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.385704 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.392405 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.385805 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpv57\" (UniqueName: \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.502116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.501994 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.502116 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.502057 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.502398 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.502182 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpv57\" (UniqueName: \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.502937 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.502905 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.512584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.512550 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.565078 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.565051 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpv57\" (UniqueName: \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\") pod \"maas-api-revoke-sub-ff5f835a-9ndrw\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:57.587803 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:57.586522 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:55:58.287874 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:55:58.287840 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cb0cf0_7aa9_4436_a2d4_dc1c4c6ff9eb.slice/crio-4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2 WatchSource:0}: Error finding container 4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2: Status 404 returned error can't find the container with id 4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2 Nov 04 02:55:58.290027 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:58.290003 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw"] Nov 04 02:55:59.086886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:59.086777 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" event={"ID":"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb","Type":"ContainerStarted","Data":"d1ae222f06ec9b3e45b43b3a1d9dd2a93f20a4896fbb3022988185fe64bba2fb"} Nov 04 02:55:59.086886 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:59.086880 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" event={"ID":"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb","Type":"ContainerStarted","Data":"4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2"} Nov 04 02:55:59.211801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:59.211746 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" podStartSLOduration=2.211732604 podStartE2EDuration="2.211732604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:59.20543782 +0000 UTC m=+1406.171938812" watchObservedRunningTime="2026-11-04 02:55:59.211732604 +0000 UTC m=+1406.178233597" Nov 04 02:55:59.446620 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:59.446583 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:55:59.446934 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:55:59.446655 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:56:00.098492 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.098455 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" event={"ID":"80054ecc-72b0-4405-8f2c-2c8c283bc3f3","Type":"ContainerStarted","Data":"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9"} Nov 04 02:56:00.098712 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.098512 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:56:00.100584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.100538 4784 generic.go:356] "Generic (PLEG): container finished" podID="26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" containerID="d1ae222f06ec9b3e45b43b3a1d9dd2a93f20a4896fbb3022988185fe64bba2fb" exitCode=0 Nov 04 02:56:00.100744 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.100595 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" event={"ID":"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb","Type":"ContainerDied","Data":"d1ae222f06ec9b3e45b43b3a1d9dd2a93f20a4896fbb3022988185fe64bba2fb"} Nov 04 02:56:00.100863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.100838 4784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t6h4m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:56:00.100955 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.100883 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:56:00.326284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:00.326225 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" podStartSLOduration=1.833482279 podStartE2EDuration="6.326208203s" totalImagesPullingTime="4.492725924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:00.284677869 +0000 UTC m=+1407.251178860" watchObservedRunningTime="2026-11-04 02:56:00.326208203 +0000 UTC m=+1407.292709191" Nov 04 02:56:01.114038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.113559 4784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t6h4m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:56:01.114038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.113629 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:56:01.322307 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.322269 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:56:01.389400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.388173 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\" (UniqueName: \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\") pod \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " Nov 04 02:56:01.389400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.388302 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\") pod \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " Nov 04 02:56:01.389400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.388442 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\") pod \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\" (UID: \"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb\") " Nov 04 02:56:01.389400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.388765 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca" pod "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" (UID: "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:56:01.389400 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.389314 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:56:01.390888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.390860 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca" pod "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" (UID: "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:56:01.391093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.391076 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57" pod "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" (UID: "26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb"). InnerVolumeSpecName "kube-api-access-qpv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:56:01.491293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.491253 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:56:01.491293 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:01.491291 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpv57\" (UniqueName: \"kubernetes.io/projected/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb-kube-api-access-qpv57\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:56:02.123442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:02.123408 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" Nov 04 02:56:02.123442 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:02.123418 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw" event={"ID":"26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb","Type":"ContainerDied","Data":"4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2"} Nov 04 02:56:02.123919 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:02.123505 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1a31c44b9b3fc12fc069199655bfb1eed9d02325d3842168ab62cffcfc15c2" Nov 04 02:56:03.998522 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:03.998486 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:56:03.998930 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:03.998560 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:56:04.114428 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:04.114397 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 02:56:06.406041 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:56:06.406004 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:56:23.767214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:23.766304 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.190217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.189305 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.190217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.189950 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:56:24.190532 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.190234 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 02:56:24.769350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.769300 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.770423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.770398 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:56:24.964346 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.964298 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.994360 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:24.994312 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.043284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.036521 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 02:56:25.179216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.167692 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:56:25.282512 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.212741 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.291217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.212779 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 04 02:56:25.396269 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.212807 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 02:56:25.431216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.212802 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:56:25.432943 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.343237 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:56:25.496355 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.496323 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:56:25.508335 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.508300 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:56:25.547347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.428842 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.887217 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.864166 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 02:56:25.990750 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.974179 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.018309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.001461 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:56:26.020501 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:25.997291 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.036098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.020931 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:56:26.037470 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.037435 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.060210 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.038096 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:56:26.423219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.417918 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.475567 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.446027 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:56:26.566465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.505492 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 02:56:26.597365 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.594484 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.598041 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.595136 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 02:56:26.598161 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.598087 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:56:26.601157 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.601125 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 04 02:56:26.604945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.604917 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.624244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.609996 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:56:26.633506 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.581612 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Nov 04 02:56:26.692722 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.692640 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 02:56:26.699100 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.694019 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 02:56:26.699910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.699885 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 02:56:26.700724 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.700707 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 02:56:26.701020 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.700911 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 02:56:26.715125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.715057 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 02:56:26.715594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.715576 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 02:56:26.715759 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.715744 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 02:56:26.735801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:26.735587 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 02:56:33.644742 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.644709 4784 scope.go:122] "RemoveContainer" containerID="64f857a0247a8cc81f026e898b4a3c03bd9c68968b2bba3543a7ce58dbeb8968" Nov 04 02:56:33.660429 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.660405 4784 scope.go:122] "RemoveContainer" containerID="e7eb82df59730d49da66899ad492406084e22ef98e51a03e7013f9ed584eb5d3" Nov 04 02:56:33.671652 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.671631 4784 scope.go:122] "RemoveContainer" containerID="8c0b18f38cc0c983555cdbb6a758817ae6c3e953d1c7221aa35bcffd79d6ff2f" Nov 04 02:56:33.682084 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.682065 4784 scope.go:122] "RemoveContainer" containerID="ca8ffd8c6b5f7034aed2161d076d0397a8063cdf0efb69bb6bafaccd9080879f" Nov 04 02:56:33.693035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.693018 4784 scope.go:122] "RemoveContainer" containerID="b42efddce37cddb48ebc5fcfc2b009b0927e702322fcd28dc689566a59e2d271" Nov 04 02:56:33.707403 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.707381 4784 scope.go:122] "RemoveContainer" containerID="10266fd931261bfeddab0fb4799e47123d6de4b2a20433e308e1df5b8169ddac" Nov 04 02:56:33.718583 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.718560 4784 scope.go:122] "RemoveContainer" containerID="6063f7a5b14c1853c32b186a99b130efae8db8772cdd4621de751ad88fcb9b5c" Nov 04 02:56:33.729472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.729452 4784 scope.go:122] "RemoveContainer" containerID="0dc1d8f91d3262816d920e2bc97ced38d7320213320b43c7fc4101c59da00213" Nov 04 02:56:33.739698 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.739661 4784 scope.go:122] "RemoveContainer" containerID="067b28692b0619967f07891573b221ccdadc5a4f4dc330d0ecc25c958d0078b8" Nov 04 02:56:33.768486 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.768287 4784 scope.go:122] "RemoveContainer" containerID="71634faf91118bbf7583fadf0f9b1dfc80ed8a35ee12b5dee006ab752a31a062" Nov 04 02:56:33.786347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:33.786314 4784 scope.go:122] "RemoveContainer" containerID="4f2d32d43c50870332653ffe33a429c59a134c34e17695bfc2591b642068f6ec" Nov 04 02:56:36.402128 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:56:36.402087 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:56:51.779624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.779575 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww"] Nov 04 02:56:51.780361 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.780346 4784 cpu_manager.go:414] "containerMap: removing container" podUID="26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" containerName="revoke-keys" Nov 04 02:56:51.780418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.780362 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" containerName="revoke-keys" Nov 04 02:56:51.780536 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.780527 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" containerName="revoke-keys" Nov 04 02:56:51.784295 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.784278 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821378 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821339 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821469 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821517 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821549 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821546 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821726 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821764 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11de3b8-689f-4e98-b418-0a1794cf7907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821828 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821927 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821909 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdz55\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-kube-api-access-qdz55\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.821963 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.821955 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.899719 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.899683 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww"] Nov 04 02:56:51.923314 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923276 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923401 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923466 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923496 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923624 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923529 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923663 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923728 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923706 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11de3b8-689f-4e98-b418-0a1794cf7907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923782 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923844 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.923888 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.923860 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdz55\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-kube-api-access-qdz55\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.924367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.924220 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.924367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.924303 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.924367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.924311 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.924584 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.924492 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11de3b8-689f-4e98-b418-0a1794cf7907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.926367 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.926344 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.926687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.926670 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.986414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.986381 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:51.986414 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:51.986379 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdz55\" (UniqueName: \"kubernetes.io/projected/f11de3b8-689f-4e98-b418-0a1794cf7907-kube-api-access-qdz55\") pod \"maas-default-gateway-openshift-default-76597d6855-jwwww\" (UID: \"f11de3b8-689f-4e98-b418-0a1794cf7907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:52.099133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:52.099045 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:52.544087 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:56:52.544052 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11de3b8_689f_4e98_b418_0a1794cf7907.slice/crio-d8d244dbac439df25df08851067a97df5a8305a785fd1e64be0da7d9d8407c28 WatchSource:0}: Error finding container d8d244dbac439df25df08851067a97df5a8305a785fd1e64be0da7d9d8407c28: Status 404 returned error can't find the container with id d8d244dbac439df25df08851067a97df5a8305a785fd1e64be0da7d9d8407c28 Nov 04 02:56:52.550573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:52.549520 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww"] Nov 04 02:56:52.559034 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:52.558983 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:56:52.559159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:52.559089 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:56:52.559159 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:52.559138 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:56:53.518847 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:53.518808 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" event={"ID":"f11de3b8-689f-4e98-b418-0a1794cf7907","Type":"ContainerStarted","Data":"57c26a66713e8df953a8c47227054f3c10aef9519a0abd3ae9023b559eb42125"} Nov 04 02:56:53.519407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:53.518862 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" event={"ID":"f11de3b8-689f-4e98-b418-0a1794cf7907","Type":"ContainerStarted","Data":"d8d244dbac439df25df08851067a97df5a8305a785fd1e64be0da7d9d8407c28"} Nov 04 02:56:53.752416 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:53.752365 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podStartSLOduration=2.752349314 podStartE2EDuration="2.752349314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:53.699730719 +0000 UTC m=+1460.666231711" watchObservedRunningTime="2026-11-04 02:56:53.752349314 +0000 UTC m=+1460.718850305" Nov 04 02:56:54.099876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:54.099836 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:56:54.101563 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:54.101536 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:54.101697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:54.101585 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:56:55.099561 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:55.099527 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:55.099990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:55.099576 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:56:56.100048 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:56.100007 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:56.100633 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:56.100083 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:56:57.100315 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:57.100278 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:57.101236 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:57.100354 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:56:58.100062 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:58.100021 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:58.100285 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:58.100077 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:56:59.100412 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:59.100371 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:56:59.100846 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:56:59.100442 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:57:00.099646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:00.099606 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 02:57:00.099835 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:00.099675 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 02:57:01.412649 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:01.412612 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:57:01.413222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:01.413166 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:57:01.414284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:01.414261 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" Nov 04 02:57:06.403095 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:57:06.403052 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:57:19.059031 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.058994 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw"] Nov 04 02:57:19.109359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.109320 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613a3e29-j2btw"] Nov 04 02:57:19.165305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.165273 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99776a70-617e-4514-af4e-feb71ebcef8f" path="/var/lib/kubelet/pods/99776a70-617e-4514-af4e-feb71ebcef8f/volumes" Nov 04 02:57:19.878127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.878093 4784 generic.go:356] "Generic (PLEG): container finished" podID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerID="7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253" exitCode=0 Nov 04 02:57:19.878387 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.878177 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerDied","Data":"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253"} Nov 04 02:57:19.878779 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.878760 4784 scope.go:122] "RemoveContainer" containerID="7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253" Nov 04 02:57:19.880665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.880625 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:57:19.880791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.880704 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:57:19.880791 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:19.880744 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:57:20.892389 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:20.892353 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerStarted","Data":"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab"} Nov 04 02:57:21.064214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:21.062537 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:57:21.082406 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:21.082359 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:21.082697 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:21.082447 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:21.484906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:21.484864 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:57:22.075681 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:22.075640 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:22.076350 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:22.076315 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:23.073216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:23.067592 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:23.073216 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:23.067654 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:24.078668 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:24.078623 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:24.081215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:24.079303 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:25.073541 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.069780 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:25.076438 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.076392 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:25.320038 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.319997 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d"] Nov 04 02:57:25.323614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.323569 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.347029 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.346992 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.347393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.347365 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.347393 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.347392 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5n6\" (UniqueName: \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.388566 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.388533 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d"] Nov 04 02:57:25.426012 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.425976 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:57:25.426233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.426063 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:57:25.426233 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.426144 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:57:25.426379 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.425997 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:57:25.449261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.449219 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.449261 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.449263 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5n6\" (UniqueName: \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.449544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.449389 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.450165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.450146 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.452058 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.452032 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.512287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.512255 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5n6\" (UniqueName: \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\") pod \"maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:25.637914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:25.637832 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:26.081783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:26.076631 4784 patch_prober.go:29] interesting pod/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 02:57:26.081783 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:26.076686 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 02:57:26.085255 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:57:26.085170 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4b8781_cd9d_4631_b0d9_ae4587055c9d.slice/crio-0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e WatchSource:0}: Error finding container 0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e: Status 404 returned error can't find the container with id 0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e Nov 04 02:57:26.088444 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:26.088417 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d"] Nov 04 02:57:26.955022 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:26.954983 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" event={"ID":"ad4b8781-cd9d-4631-b0d9-ae4587055c9d","Type":"ContainerStarted","Data":"bb4e3b95aa5a1531ca3199012be2f689f0ab6d802c0a670a12126e752b23ae68"} Nov 04 02:57:26.955544 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:26.955047 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" event={"ID":"ad4b8781-cd9d-4631-b0d9-ae4587055c9d","Type":"ContainerStarted","Data":"0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e"} Nov 04 02:57:27.063183 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:27.063153 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:57:27.064284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:27.064266 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:57:27.078594 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:27.078545 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" podStartSLOduration=2.078530568 podStartE2EDuration="2.078530568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:57:27.072322979 +0000 UTC m=+1494.038824023" watchObservedRunningTime="2026-11-04 02:57:27.078530568 +0000 UTC m=+1494.045031560" Nov 04 02:57:27.965765 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:27.965731 4784 generic.go:356] "Generic (PLEG): container finished" podID="ad4b8781-cd9d-4631-b0d9-ae4587055c9d" containerID="bb4e3b95aa5a1531ca3199012be2f689f0ab6d802c0a670a12126e752b23ae68" exitCode=0 Nov 04 02:57:27.966277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:27.965873 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" event={"ID":"ad4b8781-cd9d-4631-b0d9-ae4587055c9d","Type":"ContainerDied","Data":"bb4e3b95aa5a1531ca3199012be2f689f0ab6d802c0a670a12126e752b23ae68"} Nov 04 02:57:29.168408 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.168378 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:29.249615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.249530 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\") pod \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " Nov 04 02:57:29.249615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.249590 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\") pod \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " Nov 04 02:57:29.249615 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.249612 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\" (UniqueName: \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\") pod \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\" (UID: \"ad4b8781-cd9d-4631-b0d9-ae4587055c9d\") " Nov 04 02:57:29.249953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.249926 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca" pod "ad4b8781-cd9d-4631-b0d9-ae4587055c9d" (UID: "ad4b8781-cd9d-4631-b0d9-ae4587055c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:29.250382 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.250358 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:29.251892 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.251871 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6" pod "ad4b8781-cd9d-4631-b0d9-ae4587055c9d" (UID: "ad4b8781-cd9d-4631-b0d9-ae4587055c9d"). InnerVolumeSpecName "kube-api-access-rv5n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:29.251994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.251947 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca" pod "ad4b8781-cd9d-4631-b0d9-ae4587055c9d" (UID: "ad4b8781-cd9d-4631-b0d9-ae4587055c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:57:29.351923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.351887 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:29.351923 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.351920 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5n6\" (UniqueName: \"kubernetes.io/projected/ad4b8781-cd9d-4631-b0d9-ae4587055c9d-kube-api-access-rv5n6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:29.990259 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.990219 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" event={"ID":"ad4b8781-cd9d-4631-b0d9-ae4587055c9d","Type":"ContainerDied","Data":"0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e"} Nov 04 02:57:29.990471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.990283 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a00c72e39172b19acf1c927d6aaf2e276884430b747a749ae4ac6d46b15fa9e" Nov 04 02:57:29.990471 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:29.990235 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d" Nov 04 02:57:33.588114 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:33.588085 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:57:33.588658 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:33.588089 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 02:57:33.594515 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:33.594493 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:57:33.594669 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:33.594494 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 02:57:34.052291 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:34.052257 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27"] Nov 04 02:57:34.102974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:34.102937 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9c3cb5-zqc27"] Nov 04 02:57:35.171423 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:35.171380 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade1dec9-f5ed-479e-a37b-67aaeea9d16c" path="/var/lib/kubelet/pods/ade1dec9-f5ed-479e-a37b-67aaeea9d16c/volumes" Nov 04 02:57:36.408727 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:57:36.408686 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:57:37.498966 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:37.498923 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:57:37.582303 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:37.582262 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:57:38.903740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.903704 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4"] Nov 04 02:57:38.904489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.904473 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ad4b8781-cd9d-4631-b0d9-ae4587055c9d" containerName="revoke-keys" Nov 04 02:57:38.904489 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.904489 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4b8781-cd9d-4631-b0d9-ae4587055c9d" containerName="revoke-keys" Nov 04 02:57:38.904695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.904683 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4b8781-cd9d-4631-b0d9-ae4587055c9d" containerName="revoke-keys" Nov 04 02:57:38.908179 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.908160 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:38.972219 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.971683 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:57:38.974220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.972485 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:57:38.974220 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.972556 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:57:38.978592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.977954 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:38.978592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.978076 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:38.978592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.978250 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7857v\" (UniqueName: \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:38.992215 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:38.987666 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4"] Nov 04 02:57:39.022243 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.020697 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:57:39.125909 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.125868 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.126309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.126283 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.126629 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.126608 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7857v\" (UniqueName: \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.128234 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.128118 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.132276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.132166 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.207276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.203373 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7857v\" (UniqueName: \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\") pod \"maas-api-revoke-sub-4b0969dd-hlcw4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.229040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.228885 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:39.842676 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:57:39.842645 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ef2e9_e467_48d0_b381_3aaf5eabcbc4.slice/crio-82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c WatchSource:0}: Error finding container 82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c: Status 404 returned error can't find the container with id 82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c Nov 04 02:57:39.846091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.846070 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4"] Nov 04 02:57:39.866655 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.866619 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd"] Nov 04 02:57:39.871555 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.871526 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:39.933639 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.933594 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:39.934123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.933725 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:39.934123 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.933805 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fkf\" (UniqueName: \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:39.956640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:39.956597 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd"] Nov 04 02:57:40.035780 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.035741 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.035971 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.035809 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.035971 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.035875 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fkf\" (UniqueName: \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.036488 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.036465 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.057238 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.057204 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.098884 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.098806 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fkf\" (UniqueName: \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\") pod \"maas-api-revoke-keys-e2e-ait-87191e54-4ctkd\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.202338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.202296 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:40.207769 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.207731 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" event={"ID":"452ef2e9-e467-48d0-b381-3aaf5eabcbc4","Type":"ContainerStarted","Data":"82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c"} Nov 04 02:57:40.801717 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:57:40.801679 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08704987_25eb_4193_b03d_32ae14765923.slice/crio-c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4 WatchSource:0}: Error finding container c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4: Status 404 returned error can't find the container with id c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4 Nov 04 02:57:40.805093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:40.804997 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd"] Nov 04 02:57:41.218959 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:41.218916 4784 generic.go:356] "Generic (PLEG): container finished" podID="452ef2e9-e467-48d0-b381-3aaf5eabcbc4" containerID="966bb8f0202ba08e4f61b571e776ec7749b5448fed1fc9da0865e7752d6e2d79" exitCode=0 Nov 04 02:57:41.219482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:41.218972 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" event={"ID":"452ef2e9-e467-48d0-b381-3aaf5eabcbc4","Type":"ContainerDied","Data":"966bb8f0202ba08e4f61b571e776ec7749b5448fed1fc9da0865e7752d6e2d79"} Nov 04 02:57:41.221593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:41.221496 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" event={"ID":"08704987-25eb-4193-b03d-32ae14765923","Type":"ContainerStarted","Data":"9ce57f40cfb385248840701569b11dd9881b4ff1f76acdb6974ea65333491210"} Nov 04 02:57:41.221593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:41.221561 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" event={"ID":"08704987-25eb-4193-b03d-32ae14765923","Type":"ContainerStarted","Data":"c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4"} Nov 04 02:57:41.460370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:41.460319 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" podStartSLOduration=2.460304154 podStartE2EDuration="2.460304154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:57:41.458454064 +0000 UTC m=+1508.424955057" watchObservedRunningTime="2026-11-04 02:57:41.460304154 +0000 UTC m=+1508.426805182" Nov 04 02:57:42.232870 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.232778 4784 generic.go:356] "Generic (PLEG): container finished" podID="08704987-25eb-4193-b03d-32ae14765923" containerID="9ce57f40cfb385248840701569b11dd9881b4ff1f76acdb6974ea65333491210" exitCode=0 Nov 04 02:57:42.233352 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.232863 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" event={"ID":"08704987-25eb-4193-b03d-32ae14765923","Type":"ContainerDied","Data":"9ce57f40cfb385248840701569b11dd9881b4ff1f76acdb6974ea65333491210"} Nov 04 02:57:42.433337 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.433307 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:42.500113 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.500013 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" podUID="02bc2e69-a1ec-403c-af02-79053de37dd2" containerName="payload-processing" containerID="cri-o://1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13" gracePeriod=25 Nov 04 02:57:42.511560 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.511530 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\") pod \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " Nov 04 02:57:42.511751 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.511731 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\" (UniqueName: \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\") pod \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " Nov 04 02:57:42.511921 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.511902 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\") pod \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\" (UID: \"452ef2e9-e467-48d0-b381-3aaf5eabcbc4\") " Nov 04 02:57:42.511998 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.511949 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca" pod "452ef2e9-e467-48d0-b381-3aaf5eabcbc4" (UID: "452ef2e9-e467-48d0-b381-3aaf5eabcbc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:42.512610 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.512586 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:42.514083 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.514060 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v" pod "452ef2e9-e467-48d0-b381-3aaf5eabcbc4" (UID: "452ef2e9-e467-48d0-b381-3aaf5eabcbc4"). InnerVolumeSpecName "kube-api-access-7857v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:42.514593 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.514570 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca" pod "452ef2e9-e467-48d0-b381-3aaf5eabcbc4" (UID: "452ef2e9-e467-48d0-b381-3aaf5eabcbc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:57:42.583055 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.583004 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" podUID="135614f9-124c-41c2-b7c5-b579e82ae0d5" containerName="payload-pre-processing" containerID="cri-o://d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667" gracePeriod=25 Nov 04 02:57:42.614902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.614871 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7857v\" (UniqueName: \"kubernetes.io/projected/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-kube-api-access-7857v\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:42.615065 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.614906 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452ef2e9-e467-48d0-b381-3aaf5eabcbc4-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:42.876077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.876044 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:57:42.925994 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:42.925956 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:57:43.023614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.023524 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\" (UniqueName: \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\") pod \"135614f9-124c-41c2-b7c5-b579e82ae0d5\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " Nov 04 02:57:43.023614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.023574 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\" (UniqueName: \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\") pod \"02bc2e69-a1ec-403c-af02-79053de37dd2\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " Nov 04 02:57:43.023848 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.023674 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\") pod \"135614f9-124c-41c2-b7c5-b579e82ae0d5\" (UID: \"135614f9-124c-41c2-b7c5-b579e82ae0d5\") " Nov 04 02:57:43.023848 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.023730 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\") pod \"02bc2e69-a1ec-403c-af02-79053de37dd2\" (UID: \"02bc2e69-a1ec-403c-af02-79053de37dd2\") " Nov 04 02:57:43.024267 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.024238 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume" pod "02bc2e69-a1ec-403c-af02-79053de37dd2" (UID: "02bc2e69-a1ec-403c-af02-79053de37dd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:43.024592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.024572 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume" pod "135614f9-124c-41c2-b7c5-b579e82ae0d5" (UID: "135614f9-124c-41c2-b7c5-b579e82ae0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:43.024795 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.024772 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc2e69-a1ec-403c-af02-79053de37dd2-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.026143 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.026111 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw" pod "135614f9-124c-41c2-b7c5-b579e82ae0d5" (UID: "135614f9-124c-41c2-b7c5-b579e82ae0d5"). InnerVolumeSpecName "kube-api-access-226hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:43.026493 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.026467 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc" pod "02bc2e69-a1ec-403c-af02-79053de37dd2" (UID: "02bc2e69-a1ec-403c-af02-79053de37dd2"). InnerVolumeSpecName "kube-api-access-lmddc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:43.126851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.126809 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226hw\" (UniqueName: \"kubernetes.io/projected/135614f9-124c-41c2-b7c5-b579e82ae0d5-kube-api-access-226hw\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.126851 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.126851 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmddc\" (UniqueName: \"kubernetes.io/projected/02bc2e69-a1ec-403c-af02-79053de37dd2-kube-api-access-lmddc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.127133 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.126868 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135614f9-124c-41c2-b7c5-b579e82ae0d5-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.244091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.244050 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" Nov 04 02:57:43.244091 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.244075 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4" event={"ID":"452ef2e9-e467-48d0-b381-3aaf5eabcbc4","Type":"ContainerDied","Data":"82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c"} Nov 04 02:57:43.244596 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.244121 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d3ef22cb2f3df9545faa110f8eea1fb9e0b9f0db301f4d7ffcb6ab2b5b2f5c" Nov 04 02:57:43.245955 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.245924 4784 generic.go:356] "Generic (PLEG): container finished" podID="02bc2e69-a1ec-403c-af02-79053de37dd2" containerID="1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13" exitCode=0 Nov 04 02:57:43.246105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.245959 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" Nov 04 02:57:43.246105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.246008 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" event={"ID":"02bc2e69-a1ec-403c-af02-79053de37dd2","Type":"ContainerDied","Data":"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13"} Nov 04 02:57:43.246105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.246054 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n" event={"ID":"02bc2e69-a1ec-403c-af02-79053de37dd2","Type":"ContainerDied","Data":"985d6bd8983ef99afa95d56666f510cab7e59096cc75d2d88e6d2646f24e0eb8"} Nov 04 02:57:43.246105 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.246082 4784 scope.go:122] "RemoveContainer" containerID="1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13" Nov 04 02:57:43.248040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.248014 4784 generic.go:356] "Generic (PLEG): container finished" podID="135614f9-124c-41c2-b7c5-b579e82ae0d5" containerID="d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667" exitCode=0 Nov 04 02:57:43.248149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.248054 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" Nov 04 02:57:43.248149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.248054 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" event={"ID":"135614f9-124c-41c2-b7c5-b579e82ae0d5","Type":"ContainerDied","Data":"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667"} Nov 04 02:57:43.248149 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.248108 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4" event={"ID":"135614f9-124c-41c2-b7c5-b579e82ae0d5","Type":"ContainerDied","Data":"e1c8e7780bd7c43d57664bd9cc4cfeab48432d87fd99b1106d837e52b50e620e"} Nov 04 02:57:43.262612 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.262585 4784 scope.go:122] "RemoveContainer" containerID="1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13" Nov 04 02:57:43.263005 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:57:43.262980 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13\": container with ID starting with 1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13 not found: ID does not exist" containerID="1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13" Nov 04 02:57:43.263099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.263014 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13"} err="failed to get container status \"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13\": rpc error: code = NotFound desc = could not find container \"1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13\": container with ID starting with 1c59994e74a4021007988dac1c0cb0b0a6cd307dfaab62ad2a2a560d0ee2dc13 not found: ID does not exist" Nov 04 02:57:43.263099 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.263037 4784 scope.go:122] "RemoveContainer" containerID="d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667" Nov 04 02:57:43.275552 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.275522 4784 scope.go:122] "RemoveContainer" containerID="d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667" Nov 04 02:57:43.275943 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:57:43.275914 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667\": container with ID starting with d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667 not found: ID does not exist" containerID="d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667" Nov 04 02:57:43.276040 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.275953 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667"} err="failed to get container status \"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667\": rpc error: code = NotFound desc = could not find container \"d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667\": container with ID starting with d1b98543d98ed6c5591db39b09b8da4524a42bdeece6d6ceb1a8ee898369b667 not found: ID does not exist" Nov 04 02:57:43.421469 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.421439 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:57:43.427821 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.427795 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2c1c4f09-79984b5b5b-8bb4n"] Nov 04 02:57:43.454009 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.453978 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:43.532757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.532672 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\" (UniqueName: \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\") pod \"08704987-25eb-4193-b03d-32ae14765923\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " Nov 04 02:57:43.532902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.532759 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\") pod \"08704987-25eb-4193-b03d-32ae14765923\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " Nov 04 02:57:43.532902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.532883 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\") pod \"08704987-25eb-4193-b03d-32ae14765923\" (UID: \"08704987-25eb-4193-b03d-32ae14765923\") " Nov 04 02:57:43.533318 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.533292 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca" pod "08704987-25eb-4193-b03d-32ae14765923" (UID: "08704987-25eb-4193-b03d-32ae14765923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:43.533727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.533708 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08704987-25eb-4193-b03d-32ae14765923-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.535097 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.535076 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf" pod "08704987-25eb-4193-b03d-32ae14765923" (UID: "08704987-25eb-4193-b03d-32ae14765923"). InnerVolumeSpecName "kube-api-access-c6fkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:43.535097 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.535086 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca" pod "08704987-25eb-4193-b03d-32ae14765923" (UID: "08704987-25eb-4193-b03d-32ae14765923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:57:43.591244 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.591182 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:57:43.597396 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.597365 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2c1c4f09-8689cb4c5f-htpk4"] Nov 04 02:57:43.635115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.635080 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fkf\" (UniqueName: \"kubernetes.io/projected/08704987-25eb-4193-b03d-32ae14765923-kube-api-access-c6fkf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:43.635115 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:43.635114 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08704987-25eb-4193-b03d-32ae14765923-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:44.262824 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:44.262789 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" event={"ID":"08704987-25eb-4193-b03d-32ae14765923","Type":"ContainerDied","Data":"c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4"} Nov 04 02:57:44.263338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:44.262849 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd" Nov 04 02:57:44.263338 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:44.262853 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8072df7c42776b79cdc390a35ee05834003dedda2ddb3616263e0b2980c64a4" Nov 04 02:57:45.175689 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:45.175656 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bc2e69-a1ec-403c-af02-79053de37dd2" path="/var/lib/kubelet/pods/02bc2e69-a1ec-403c-af02-79053de37dd2/volumes" Nov 04 02:57:45.176121 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:45.176107 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135614f9-124c-41c2-b7c5-b579e82ae0d5" path="/var/lib/kubelet/pods/135614f9-124c-41c2-b7c5-b579e82ae0d5/volumes" Nov 04 02:57:55.054165 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:55.054132 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47"] Nov 04 02:57:55.103095 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:55.103060 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88cd5873-47n47"] Nov 04 02:57:55.163013 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:57:55.162979 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa568d9-644f-4c10-b3af-502c3effc8d2" path="/var/lib/kubelet/pods/0aa568d9-644f-4c10-b3af-502c3effc8d2/volumes" Nov 04 02:58:06.052323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:06.052285 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v"] Nov 04 02:58:06.103156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:06.103119 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6e80c-2b98v"] Nov 04 02:58:06.401991 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:58:06.401895 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:58:07.162619 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:07.162585 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5db053-85b8-468a-b622-c1866ed0d8dd" path="/var/lib/kubelet/pods/1b5db053-85b8-468a-b622-c1866ed0d8dd/volumes" Nov 04 02:58:08.484736 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.484698 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7"] Nov 04 02:58:08.485455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485441 4784 cpu_manager.go:414] "containerMap: removing container" podUID="135614f9-124c-41c2-b7c5-b579e82ae0d5" containerName="payload-pre-processing" Nov 04 02:58:08.485455 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485456 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135614f9-124c-41c2-b7c5-b579e82ae0d5" containerName="payload-pre-processing" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485493 4784 cpu_manager.go:414] "containerMap: removing container" podUID="02bc2e69-a1ec-403c-af02-79053de37dd2" containerName="payload-processing" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485500 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bc2e69-a1ec-403c-af02-79053de37dd2" containerName="payload-processing" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485519 4784 cpu_manager.go:414] "containerMap: removing container" podUID="08704987-25eb-4193-b03d-32ae14765923" containerName="revoke-keys" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485524 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08704987-25eb-4193-b03d-32ae14765923" containerName="revoke-keys" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485539 4784 cpu_manager.go:414] "containerMap: removing container" podUID="452ef2e9-e467-48d0-b381-3aaf5eabcbc4" containerName="revoke-keys" Nov 04 02:58:08.485547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485544 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452ef2e9-e467-48d0-b381-3aaf5eabcbc4" containerName="revoke-keys" Nov 04 02:58:08.485753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485650 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bc2e69-a1ec-403c-af02-79053de37dd2" containerName="payload-processing" Nov 04 02:58:08.485753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485667 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="08704987-25eb-4193-b03d-32ae14765923" containerName="revoke-keys" Nov 04 02:58:08.485753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485675 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="452ef2e9-e467-48d0-b381-3aaf5eabcbc4" containerName="revoke-keys" Nov 04 02:58:08.485753 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.485682 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="135614f9-124c-41c2-b7c5-b579e82ae0d5" containerName="payload-pre-processing" Nov 04 02:58:08.489397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.489367 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.547617 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.547578 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7"] Nov 04 02:58:08.587665 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.587630 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.587849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.587783 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.587902 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.587853 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69hr\" (UniqueName: \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.590229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.590123 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:58:08.590229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.590136 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:58:08.590229 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.590165 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:58:08.590495 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.590233 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:58:08.689699 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.689661 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.689906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.689763 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.689906 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.689837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69hr\" (UniqueName: \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.690456 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.690434 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.692171 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.692147 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.752402 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.752324 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69hr\" (UniqueName: \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\") pod \"maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:08.803651 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:08.803614 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:09.192402 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:58:09.192360 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c9bb61_f51c_4ed7_a651_ab5f01a0ecf1.slice/crio-12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88 WatchSource:0}: Error finding container 12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88: Status 404 returned error can't find the container with id 12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88 Nov 04 02:58:09.194453 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:09.194429 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7"] Nov 04 02:58:09.529910 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:09.529871 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" event={"ID":"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1","Type":"ContainerStarted","Data":"4903b4d910e34edd98d6f9c462c2f6427a552ac5203596799e79cd28b44ab452"} Nov 04 02:58:09.530345 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:09.529937 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" event={"ID":"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1","Type":"ContainerStarted","Data":"12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88"} Nov 04 02:58:09.652713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:09.652654 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" podStartSLOduration=1.65263089 podStartE2EDuration="1.65263089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:09.646503343 +0000 UTC m=+1536.613004336" watchObservedRunningTime="2026-11-04 02:58:09.65263089 +0000 UTC m=+1536.619131968" Nov 04 02:58:10.541032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:10.540996 4784 generic.go:356] "Generic (PLEG): container finished" podID="52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" containerID="4903b4d910e34edd98d6f9c462c2f6427a552ac5203596799e79cd28b44ab452" exitCode=0 Nov 04 02:58:10.541500 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:10.541125 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" event={"ID":"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1","Type":"ContainerDied","Data":"4903b4d910e34edd98d6f9c462c2f6427a552ac5203596799e79cd28b44ab452"} Nov 04 02:58:11.730907 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.730879 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:11.868176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.868050 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\" (UniqueName: \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\") pod \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " Nov 04 02:58:11.868383 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.868351 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\") pod \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " Nov 04 02:58:11.868470 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.868448 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\") pod \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\" (UID: \"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1\") " Nov 04 02:58:11.868915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.868886 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca" pod "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" (UID: "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:11.869323 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.869298 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:11.870997 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.870972 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca" pod "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" (UID: "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:58:11.871673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.871654 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr" pod "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" (UID: "52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1"). InnerVolumeSpecName "kube-api-access-h69hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:11.971158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.971107 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69hr\" (UniqueName: \"kubernetes.io/projected/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-kube-api-access-h69hr\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:11.971158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:11.971154 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:12.560880 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:12.560846 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" Nov 04 02:58:12.561092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:12.560878 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7" event={"ID":"52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1","Type":"ContainerDied","Data":"12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88"} Nov 04 02:58:12.561092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:12.560927 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ea55e3730229af9aea69cd97fb6caf4603e218e8e9dcf250ff834146c69a88" Nov 04 02:58:16.636693 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:16.636659 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d"] Nov 04 02:58:16.695845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:16.692305 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2c1c4f09-mvl6d"] Nov 04 02:58:16.704550 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:16.704506 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:58:16.704830 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:16.704763 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" containerID="cri-o://24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab" gracePeriod=30 Nov 04 02:58:17.162952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:17.162916 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4b8781-cd9d-4631-b0d9-ae4587055c9d" path="/var/lib/kubelet/pods/ad4b8781-cd9d-4631-b0d9-ae4587055c9d/volumes" Nov 04 02:58:22.110337 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.110306 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.224716 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.224773 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.224806 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.224910 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.225053 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.225140 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.225274 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.225355 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.225687 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.225392 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\") pod \"9dec8ab1-099a-4e1b-b525-2a430f676947\" (UID: \"9dec8ab1-099a-4e1b-b525-2a430f676947\") " Nov 04 02:58:22.229727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.229475 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:22.229727 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.229534 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:22.229950 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.229742 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:22.230213 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.230081 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:22.230754 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.230727 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:22.283053 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.280397 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "kube-api-access-7g8dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:22.283671 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.283617 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:22.290969 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.290922 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:58:22.291151 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.291083 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token" pod "9dec8ab1-099a-4e1b-b525-2a430f676947" (UID: "9dec8ab1-099a-4e1b-b525-2a430f676947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328576 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328614 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328628 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328644 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328659 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328672 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dec8ab1-099a-4e1b-b525-2a430f676947-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328685 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328701 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dec8ab1-099a-4e1b-b525-2a430f676947-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.328792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.328714 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8dv\" (UniqueName: \"kubernetes.io/projected/9dec8ab1-099a-4e1b-b525-2a430f676947-kube-api-access-7g8dv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:22.818093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.818054 4784 generic.go:356] "Generic (PLEG): container finished" podID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerID="24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab" exitCode=0 Nov 04 02:58:22.818289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.818151 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" Nov 04 02:58:22.818289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.818166 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerDied","Data":"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab"} Nov 04 02:58:22.818289 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.818279 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v" event={"ID":"9dec8ab1-099a-4e1b-b525-2a430f676947","Type":"ContainerDied","Data":"cd686dee959bdef23209a8c30e52aa166755a06e25a743487831d9df4619c372"} Nov 04 02:58:22.818472 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.818315 4784 scope.go:122] "RemoveContainer" containerID="24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab" Nov 04 02:58:22.831465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.831447 4784 scope.go:122] "RemoveContainer" containerID="7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253" Nov 04 02:58:22.843093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.843070 4784 scope.go:122] "RemoveContainer" containerID="24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab" Nov 04 02:58:22.843447 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:58:22.843428 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab\": container with ID starting with 24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab not found: ID does not exist" containerID="24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab" Nov 04 02:58:22.843529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.843460 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab"} err="failed to get container status \"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab\": rpc error: code = NotFound desc = could not find container \"24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab\": container with ID starting with 24d572cd443b352c7c324e23ed5d8486fc63a69bf5b2d1601879258e8d0144ab not found: ID does not exist" Nov 04 02:58:22.843529 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.843485 4784 scope.go:122] "RemoveContainer" containerID="7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253" Nov 04 02:58:22.843755 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:58:22.843737 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253\": container with ID starting with 7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253 not found: ID does not exist" containerID="7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253" Nov 04 02:58:22.843802 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:22.843765 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253"} err="failed to get container status \"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253\": rpc error: code = NotFound desc = could not find container \"7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253\": container with ID starting with 7e7399231f300f3696062875aeb1083c4f56fb15390264f621b006a30a14a253 not found: ID does not exist" Nov 04 02:58:23.045763 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:23.045728 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:58:23.052142 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:23.052111 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2c1c4f09-gw-openshift-default-648b465cc-mr59v"] Nov 04 02:58:23.177370 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:23.177340 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" path="/var/lib/kubelet/pods/9dec8ab1-099a-4e1b-b525-2a430f676947/volumes" Nov 04 02:58:23.189074 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:23.189028 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd"] Nov 04 02:58:23.210349 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:23.210306 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87191e54-4ctkd"] Nov 04 02:58:24.052419 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:24.052387 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss"] Nov 04 02:58:24.058673 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:24.058646 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5928b909-hlbss"] Nov 04 02:58:25.162715 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.162680 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08704987-25eb-4193-b03d-32ae14765923" path="/var/lib/kubelet/pods/08704987-25eb-4193-b03d-32ae14765923/volumes" Nov 04 02:58:25.163354 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.163337 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41124fed-a849-4f3f-827b-9fbd3ac92d70" path="/var/lib/kubelet/pods/41124fed-a849-4f3f-827b-9fbd3ac92d70/volumes" Nov 04 02:58:25.190198 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190164 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 02:58:25.190924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190907 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:25.190924 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190925 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:25.191045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190957 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:25.191045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190963 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:25.191045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190984 4784 cpu_manager.go:414] "containerMap: removing container" podUID="52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" containerName="revoke-keys" Nov 04 02:58:25.191045 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.190989 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" containerName="revoke-keys" Nov 04 02:58:25.191176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.191091 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:25.191176 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.191106 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" containerName="revoke-keys" Nov 04 02:58:25.196283 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.196264 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.254723 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.254663 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5fde04ee-openshift-default-dockercfg-hx7xr\"" Nov 04 02:58:25.287792 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.287600 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.287993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.287824 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.287993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.287849 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.287993 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.287870 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.288128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.288047 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.288128 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.288107 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.288284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.288170 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.288347 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.288322 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.288439 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.288377 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.315391 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.315331 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 02:58:25.390007 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.389942 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390061 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390125 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390158 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390239 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390206 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390456 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390546 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390476 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390568 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390606 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390578 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390663 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.390845 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.390822 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.391043 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.391013 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.391156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.391056 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.391231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.391208 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.392881 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.392856 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.393097 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.393078 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.452614 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.452580 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.452778 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.452690 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\") pod \"e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:25.510831 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:25.510724 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:26.094883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.094787 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 02:58:26.133572 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:58:26.133526 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e24565_e859_4c16_a564_051c809fff3d.slice/crio-43a6ad03cbecd78bfc377413e9628f46d1c7d709bf27a826fdfd1c149c3870b5 WatchSource:0}: Error finding container 43a6ad03cbecd78bfc377413e9628f46d1c7d709bf27a826fdfd1c149c3870b5: Status 404 returned error can't find the container with id 43a6ad03cbecd78bfc377413e9628f46d1c7d709bf27a826fdfd1c149c3870b5 Nov 04 02:58:26.137156 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.137111 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:26.137266 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.137233 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:26.137309 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.137282 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:26.875480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.875384 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerStarted","Data":"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c"} Nov 04 02:58:26.875480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:26.875448 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerStarted","Data":"43a6ad03cbecd78bfc377413e9628f46d1c7d709bf27a826fdfd1c149c3870b5"} Nov 04 02:58:27.050573 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:27.050514 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podStartSLOduration=2.050497781 podStartE2EDuration="2.050497781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:27.049115514 +0000 UTC m=+1554.015616534" watchObservedRunningTime="2026-11-04 02:58:27.050497781 +0000 UTC m=+1554.016998772" Nov 04 02:58:27.585990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:27.585955 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:27.592222 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:27.592178 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:27.885068 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:27.884981 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:27.885960 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:27.885938 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:58:33.947073 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:33.947041 4784 scope.go:122] "RemoveContainer" containerID="241910685bce8f5d5b5c7baa72211124c2a1f4b659c8975dac39cb1905d88bc5" Nov 04 02:58:33.959755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:33.959734 4784 scope.go:122] "RemoveContainer" containerID="1507bf897243200d93efcb97691fa1e4e3fb84fa3191dad4c0ab64852f1b1962" Nov 04 02:58:36.402230 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:58:36.402170 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:58:37.665353 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.665318 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 02:58:37.666180 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.666164 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dec8ab1-099a-4e1b-b525-2a430f676947" containerName="istio-proxy" Nov 04 02:58:37.671917 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.671898 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782098 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782062 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782296 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782250 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782418 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782398 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782503 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782440 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrwf\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782646 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782625 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782741 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782712 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782801 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782783 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782856 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782833 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.782914 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.782868 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.788237 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.788209 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 02:58:37.884581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884537 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.884789 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884645 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.884789 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884714 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.884789 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884759 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884792 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884855 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884952 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.884959 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.885097 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.885162 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrwf\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.885600 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.885836 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.886264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.886045 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.887684 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.886881 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.892491 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.892290 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.893797 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.893773 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.948812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.948728 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.948812 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.948735 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrwf\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\") pod \"maas-default-gateway-openshift-default-76597d6855-xr2tj\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:37.986883 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:37.986843 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:38.430456 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:58:38.430424 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8040dd82_b66b_4c36_ad01_c466a63ffc11.slice/crio-547f6d0a34bc8a77737c246a5999999295476fbb83e7fd9e2d9287b63b0643ae WatchSource:0}: Error finding container 547f6d0a34bc8a77737c246a5999999295476fbb83e7fd9e2d9287b63b0643ae: Status 404 returned error can't find the container with id 547f6d0a34bc8a77737c246a5999999295476fbb83e7fd9e2d9287b63b0643ae Nov 04 02:58:38.433033 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:38.433011 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 02:58:38.433300 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:38.433269 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:38.433364 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:38.433332 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:38.433397 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:38.433362 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:58:39.009429 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.009394 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" event={"ID":"8040dd82-b66b-4c36-ad01-c466a63ffc11","Type":"ContainerStarted","Data":"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4"} Nov 04 02:58:39.009904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.009460 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" event={"ID":"8040dd82-b66b-4c36-ad01-c466a63ffc11","Type":"ContainerStarted","Data":"547f6d0a34bc8a77737c246a5999999295476fbb83e7fd9e2d9287b63b0643ae"} Nov 04 02:58:39.137001 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.136949 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podStartSLOduration=2.136934378 podStartE2EDuration="2.136934378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:39.135995134 +0000 UTC m=+1566.102496127" watchObservedRunningTime="2026-11-04 02:58:39.136934378 +0000 UTC m=+1566.103435369" Nov 04 02:58:39.801858 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.801827 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7"] Nov 04 02:58:39.806945 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.806917 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:39.895331 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.895282 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7"] Nov 04 02:58:39.922987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.922953 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:39.923224 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.923163 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:39.923320 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.923263 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknvl\" (UniqueName: \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:39.928867 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.928839 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:58:39.929004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.928839 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:58:39.929004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.928892 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 02:58:39.929004 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.928917 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:58:39.987530 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.987501 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:39.988990 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.988967 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:39.989088 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:39.989032 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:40.025482 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.025447 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.025915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.025632 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.025915 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.025696 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknvl\" (UniqueName: \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.026231 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.026211 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.028152 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.028124 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.054572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.054476 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz"] Nov 04 02:58:40.096784 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.096747 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d602a9d-g7mgz"] Nov 04 02:58:40.099553 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.099507 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknvl\" (UniqueName: \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\") pod \"maas-api-revoke-sub-c7c5d4d3-kmrk7\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.128047 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.128021 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:40.542121 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:58:40.542079 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f24919a_96b5_42fd_8109_0b59e34905ef.slice/crio-5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d WatchSource:0}: Error finding container 5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d: Status 404 returned error can't find the container with id 5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d Nov 04 02:58:40.544602 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.544578 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7"] Nov 04 02:58:40.987692 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.987649 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:40.987876 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:40.987715 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:41.032359 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:41.032322 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" event={"ID":"9f24919a-96b5-42fd-8109-0b59e34905ef","Type":"ContainerStarted","Data":"5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d"} Nov 04 02:58:41.188737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:41.188700 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33db7759-85cd-4888-a49b-59a2b5aef2ee" path="/var/lib/kubelet/pods/33db7759-85cd-4888-a49b-59a2b5aef2ee/volumes" Nov 04 02:58:41.987757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:41.987679 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:41.987757 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:41.987743 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:42.044904 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:42.044815 4784 generic.go:356] "Generic (PLEG): container finished" podID="9f24919a-96b5-42fd-8109-0b59e34905ef" containerID="e34625270b92865ecd3802aff46e11d4bc5d49a0d2677177568970aab80713b3" exitCode=0 Nov 04 02:58:42.045306 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:42.044908 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" event={"ID":"9f24919a-96b5-42fd-8109-0b59e34905ef","Type":"ContainerDied","Data":"e34625270b92865ecd3802aff46e11d4bc5d49a0d2677177568970aab80713b3"} Nov 04 02:58:42.987466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:42.987431 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:42.987662 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:42.987488 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:43.306287 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.306261 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:43.408884 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.408839 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\" (UniqueName: \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\") pod \"9f24919a-96b5-42fd-8109-0b59e34905ef\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " Nov 04 02:58:43.409093 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.409035 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\") pod \"9f24919a-96b5-42fd-8109-0b59e34905ef\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " Nov 04 02:58:43.409169 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.409123 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\") pod \"9f24919a-96b5-42fd-8109-0b59e34905ef\" (UID: \"9f24919a-96b5-42fd-8109-0b59e34905ef\") " Nov 04 02:58:43.409435 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.409410 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca" pod "9f24919a-96b5-42fd-8109-0b59e34905ef" (UID: "9f24919a-96b5-42fd-8109-0b59e34905ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:43.410481 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.410453 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:43.414200 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.411357 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl" pod "9f24919a-96b5-42fd-8109-0b59e34905ef" (UID: "9f24919a-96b5-42fd-8109-0b59e34905ef"). InnerVolumeSpecName "kube-api-access-pknvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:43.414200 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.411544 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca" pod "9f24919a-96b5-42fd-8109-0b59e34905ef" (UID: "9f24919a-96b5-42fd-8109-0b59e34905ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:58:43.512527 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.512439 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f24919a-96b5-42fd-8109-0b59e34905ef-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:43.512527 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.512473 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pknvl\" (UniqueName: \"kubernetes.io/projected/9f24919a-96b5-42fd-8109-0b59e34905ef-kube-api-access-pknvl\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:43.987317 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.987263 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:43.987533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:43.987338 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:44.089608 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:44.089572 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" Nov 04 02:58:44.089798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:44.089606 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7" event={"ID":"9f24919a-96b5-42fd-8109-0b59e34905ef","Type":"ContainerDied","Data":"5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d"} Nov 04 02:58:44.089798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:44.089696 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7ec112e4c54832ae6e85a8e0b6b6630a5d83ba48db134f1f2f94486cc3e04d" Nov 04 02:58:44.988003 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:44.987970 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:44.988534 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:44.988030 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:45.987572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:45.987536 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 02:58:45.987770 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:45.987603 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 02:58:47.334294 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:47.334254 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:47.334996 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:47.334974 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:47.335839 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:47.335818 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 02:58:49.970496 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:49.970456 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7"] Nov 04 02:58:50.019592 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:50.019556 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f73e00ed-qmnc7"] Nov 04 02:58:51.164166 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:58:51.164132 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1" path="/var/lib/kubelet/pods/52c9bb61-f51c-4ed7-a651-ab5f01a0ecf1/volumes" Nov 04 02:59:06.402337 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:59:06.402300 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:59:12.054480 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:12.054448 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl"] Nov 04 02:59:12.102127 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:12.102088 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6d834a-cfnvl"] Nov 04 02:59:13.074175 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:13.074143 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv"] Nov 04 02:59:13.125305 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:13.125257 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9522d074-q4mwv"] Nov 04 02:59:13.210227 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:13.209918 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2a451b-6b8d-404a-9439-a629987a7b59" path="/var/lib/kubelet/pods/1d2a451b-6b8d-404a-9439-a629987a7b59/volumes" Nov 04 02:59:13.211460 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:13.211349 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75b5dee-2c99-418a-a4c7-e5a52637559f" path="/var/lib/kubelet/pods/c75b5dee-2c99-418a-a4c7-e5a52637559f/volumes" Nov 04 02:59:21.051725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.051687 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 02:59:21.052952 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.052925 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9f24919a-96b5-42fd-8109-0b59e34905ef" containerName="revoke-keys" Nov 04 02:59:21.053024 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.052956 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f24919a-96b5-42fd-8109-0b59e34905ef" containerName="revoke-keys" Nov 04 02:59:21.053240 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.053222 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f24919a-96b5-42fd-8109-0b59e34905ef" containerName="revoke-keys" Nov 04 02:59:21.058168 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.058143 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.123947 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.123902 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 02:59:21.156638 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.156596 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-db4d0787-openshift-default-dockercfg-mn4zk\"" Nov 04 02:59:21.287474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287224 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287324 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287474 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287376 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287517 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287557 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287814 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287758 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287820 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287872 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xb7\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.287981 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.287957 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.333581 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.333488 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 02:59:21.339884 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.339854 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.389154 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.389118 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 02:59:21.389740 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.389714 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.389832 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.389754 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xb7\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.389899 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.389828 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.389974 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.389958 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390032 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390004 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390087 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390040 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390087 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390075 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.390276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390119 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.390276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390230 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390276 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390267 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390511 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390448 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.390788 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.390765 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.391284 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.391228 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.391961 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.391936 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.392424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.392347 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.392424 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.392379 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.394183 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.394155 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-297806\"" Nov 04 02:59:21.395798 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.395775 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.396055 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.396034 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.408641 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.408599 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-297806-dockercfg-q6wph\"" Nov 04 02:59:21.459015 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.458980 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xb7\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.465726 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.465673 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\") pod \"e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.473535 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.473501 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 02:59:21.494256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.492769 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.494256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.492840 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.494256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.493801 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.494256 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.493934 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.529547 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.529491 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 02:59:21.606265 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.591991 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\") pod \"payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.606265 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.594473 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snp7\" (UniqueName: \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.606265 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.594713 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.707096 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.679511 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:21.707096 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.679959 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:21.747695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.743864 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.747695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.744088 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snp7\" (UniqueName: \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.747695 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.745181 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:21.847377 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:21.847330 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snp7\" (UniqueName: \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\") pod \"payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:22.123693 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.123663 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:22.565086 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:59:22.565043 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb0907e_7ded_4261_971b_6149ac28fb07.slice/crio-263c5751395e780db3a49018fab3b9cf015568b914ab2bca3a8c7a72f1d8ff86 WatchSource:0}: Error finding container 263c5751395e780db3a49018fab3b9cf015568b914ab2bca3a8c7a72f1d8ff86: Status 404 returned error can't find the container with id 263c5751395e780db3a49018fab3b9cf015568b914ab2bca3a8c7a72f1d8ff86 Nov 04 02:59:22.565918 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:59:22.565893 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c68f76e_7ba8_4aca_b778_e7b113cc460f.slice/crio-f596d30fd14c81a31e0ea24abe7a5423b0c1631676208b89eefeff4e57acc131 WatchSource:0}: Error finding container f596d30fd14c81a31e0ea24abe7a5423b0c1631676208b89eefeff4e57acc131: Status 404 returned error can't find the container with id f596d30fd14c81a31e0ea24abe7a5423b0c1631676208b89eefeff4e57acc131 Nov 04 02:59:22.568183 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.568143 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:22.568476 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.568442 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 02:59:22.569141 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.569115 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:22.569214 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.569173 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:22.574877 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.574856 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 02:59:22.629987 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.629960 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 02:59:22.632786 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:59:22.632763 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4774a3_13c9_4ca3_bd3f_59c816db2f8c.slice/crio-37387ab291c1f80d28314d5a7f6c72034b0a2ca5ac1f3fccae056f4f5f8cd757 WatchSource:0}: Error finding container 37387ab291c1f80d28314d5a7f6c72034b0a2ca5ac1f3fccae056f4f5f8cd757: Status 404 returned error can't find the container with id 37387ab291c1f80d28314d5a7f6c72034b0a2ca5ac1f3fccae056f4f5f8cd757 Nov 04 02:59:22.840734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.840628 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerStarted","Data":"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4"} Nov 04 02:59:22.840734 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.840704 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerStarted","Data":"263c5751395e780db3a49018fab3b9cf015568b914ab2bca3a8c7a72f1d8ff86"} Nov 04 02:59:22.842640 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.842608 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" event={"ID":"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c","Type":"ContainerStarted","Data":"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e"} Nov 04 02:59:22.842790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.842663 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" event={"ID":"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c","Type":"ContainerStarted","Data":"37387ab291c1f80d28314d5a7f6c72034b0a2ca5ac1f3fccae056f4f5f8cd757"} Nov 04 02:59:22.842790 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.842707 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:22.844661 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.844637 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" event={"ID":"6c68f76e-7ba8-4aca-b778-e7b113cc460f","Type":"ContainerStarted","Data":"d3a20d7e835e75ae4f0a0129390a9eb81ffdec3240a6338a8e31f4f4057015e3"} Nov 04 02:59:22.844768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.844680 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" event={"ID":"6c68f76e-7ba8-4aca-b778-e7b113cc460f","Type":"ContainerStarted","Data":"f596d30fd14c81a31e0ea24abe7a5423b0c1631676208b89eefeff4e57acc131"} Nov 04 02:59:22.844768 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.844712 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:22.968417 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:22.968311 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podStartSLOduration=2.968290674 podStartE2EDuration="2.968290674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:59:22.966331667 +0000 UTC m=+1609.932832661" watchObservedRunningTime="2026-11-04 02:59:22.968290674 +0000 UTC m=+1609.934791667" Nov 04 02:59:23.098928 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.096470 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" podStartSLOduration=2.096447045 podStartE2EDuration="2.096447045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:59:23.085713516 +0000 UTC m=+1610.052214514" watchObservedRunningTime="2026-11-04 02:59:23.096447045 +0000 UTC m=+1610.062948038" Nov 04 02:59:23.680729 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.680696 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:23.686789 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.686762 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:23.814725 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.814673 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" podStartSLOduration=2.814658071 podStartE2EDuration="2.814658071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:59:23.220827702 +0000 UTC m=+1610.187328694" watchObservedRunningTime="2026-11-04 02:59:23.814658071 +0000 UTC m=+1610.781159062" Nov 04 02:59:23.854490 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.854460 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:23.855362 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:23.855343 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 02:59:24.164270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:24.151745 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69"] Nov 04 02:59:24.164270 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:24.163076 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64713a1b-gdr69"] Nov 04 02:59:25.162875 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:25.162837 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5169994d-87fc-4bbe-acd7-328d809eefdb" path="/var/lib/kubelet/pods/5169994d-87fc-4bbe-acd7-328d809eefdb/volumes" Nov 04 02:59:28.051674 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:28.051636 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr"] Nov 04 02:59:28.102466 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:28.102432 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a4bbe8-vtdqr"] Nov 04 02:59:29.163206 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.163158 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74eb776f-a59f-4f98-ac03-cb396a461c3f" path="/var/lib/kubelet/pods/74eb776f-a59f-4f98-ac03-cb396a461c3f/volumes" Nov 04 02:59:29.782376 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.782328 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 02:59:29.791572 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.791539 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:29.849859 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.849666 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 02:59:29.858694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.855675 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld8z\" (UniqueName: \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:29.858694 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.855783 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:29.898591 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.898551 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-db4d0787\"" Nov 04 02:59:29.898848 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.898827 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-db4d0787-dockercfg-88jxn\"" Nov 04 02:59:29.958035 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.957831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:29.958737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.958710 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld8z\" (UniqueName: \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:29.973266 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:29.972059 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:30.116264 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.112445 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld8z\" (UniqueName: \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\") pod \"payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:30.169277 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.168119 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:30.564753 ip-10-0-133-209 kubenswrapper[4784]: W1104 02:59:30.564722 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aec14e_54b3_44c2_8555_9dad9a6845ce.slice/crio-562936bb8cb1f25965ccf650a0b2aa754f7c311cecaa3e92f1cc2ba42059d43d WatchSource:0}: Error finding container 562936bb8cb1f25965ccf650a0b2aa754f7c311cecaa3e92f1cc2ba42059d43d: Status 404 returned error can't find the container with id 562936bb8cb1f25965ccf650a0b2aa754f7c311cecaa3e92f1cc2ba42059d43d Nov 04 02:59:30.567705 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.567675 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 02:59:30.936077 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.936041 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" event={"ID":"f6aec14e-54b3-44c2-8555-9dad9a6845ce","Type":"ContainerStarted","Data":"38b2ab2c67b77351e81b7cc34643a38a92785e458c2ae3872072dfb653917046"} Nov 04 02:59:30.936304 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.936098 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" event={"ID":"f6aec14e-54b3-44c2-8555-9dad9a6845ce","Type":"ContainerStarted","Data":"562936bb8cb1f25965ccf650a0b2aa754f7c311cecaa3e92f1cc2ba42059d43d"} Nov 04 02:59:30.936304 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:30.936137 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:31.051672 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:31.051634 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9"] Nov 04 02:59:31.059011 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:31.058965 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050eadcc-9hpp9"] Nov 04 02:59:31.099863 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:31.099803 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" podStartSLOduration=2.099789959 podStartE2EDuration="2.099789959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:59:31.096328694 +0000 UTC m=+1618.062829687" watchObservedRunningTime="2026-11-04 02:59:31.099789959 +0000 UTC m=+1618.066290951" Nov 04 02:59:31.163351 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:31.163306 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1275091-41fd-45eb-9468-90a6fded50a8" path="/var/lib/kubelet/pods/c1275091-41fd-45eb-9468-90a6fded50a8/volumes" Nov 04 02:59:33.856953 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:33.856924 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 02:59:33.857382 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:33.857000 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 02:59:34.027407 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:34.027371 4784 scope.go:122] "RemoveContainer" containerID="923edf75f0b5417e60becd63146df039c2faa5e20de4f20101ffe800eb664cd5" Nov 04 02:59:34.041944 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:34.041920 4784 scope.go:122] "RemoveContainer" containerID="70b4b289be877e4fffa22523bbb372a0b93e0f70fcba22d2f3c35e9de8d0447d" Nov 04 02:59:34.054889 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:34.054861 4784 scope.go:122] "RemoveContainer" containerID="cb146fc302a8d7348a349f3e546d5a6c7577bd2ae17598f03f327ed136ed9654" Nov 04 02:59:36.644529 ip-10-0-133-209 kubenswrapper[4784]: E1104 02:59:36.644450 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 02:59:36.768456 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:36.768418 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:59:36.768755 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:36.768722 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:41.052855 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:41.052816 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql"] Nov 04 02:59:41.103749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:41.103707 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b97258c-j9lql"] Nov 04 02:59:41.186846 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:41.186806 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3caa815b-a11b-4d03-88ae-d2f2a4c1dbae" path="/var/lib/kubelet/pods/3caa815b-a11b-4d03-88ae-d2f2a4c1dbae/volumes" Nov 04 02:59:41.950607 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:41.950529 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 02:59:43.926384 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:43.926347 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:59:43.926804 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:43.926428 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:44.052972 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:44.052934 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f"] Nov 04 02:59:44.104125 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:44.104090 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda9d425-ts85f"] Nov 04 02:59:45.163746 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:45.163711 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75df7aec-8910-4f5e-b60f-fe0239debfc3" path="/var/lib/kubelet/pods/75df7aec-8910-4f5e-b60f-fe0239debfc3/volumes" Nov 04 02:59:46.051840 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.051755 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx"] Nov 04 02:59:46.102650 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.102616 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e506e-gxhtx"] Nov 04 02:59:46.491485 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.491453 4784 generic.go:356] "Generic (PLEG): container finished" podID="a1e24565-e859-4c16-a564-051c809fff3d" containerID="02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c" exitCode=0 Nov 04 02:59:46.491901 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.491527 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerDied","Data":"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c"} Nov 04 02:59:46.492052 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.492036 4784 scope.go:122] "RemoveContainer" containerID="02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c" Nov 04 02:59:46.493849 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.493818 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:46.493933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.493890 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:46.493933 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:46.493921 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 02:59:47.163507 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:47.163471 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383b1d22-3665-4ad5-912e-fb3afaf5f314" path="/var/lib/kubelet/pods/383b1d22-3665-4ad5-912e-fb3afaf5f314/volumes" Nov 04 02:59:47.506749 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:47.506711 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerStarted","Data":"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2"} Nov 04 02:59:47.591212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:47.586609 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:59:47.591212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:47.586740 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:47.591212 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:47.586799 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:48.052344 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:48.052313 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6"] Nov 04 02:59:48.089319 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:48.089271 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b77ddc-v8lj6"] Nov 04 02:59:48.586092 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:48.586057 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:48.586533 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:48.586126 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:49.189158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:49.189121 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0543394-456d-4170-a13f-17f51cc6674f" path="/var/lib/kubelet/pods/d0543394-456d-4170-a13f-17f51cc6674f/volumes" Nov 04 02:59:49.586761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:49.586654 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:49.586761 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:49.586719 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:50.588631 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:50.588594 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:50.589158 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:50.588655 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:51.585823 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:51.585790 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:51.586030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:51.585852 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:52.051940 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:52.051905 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh"] Nov 04 02:59:52.102465 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:52.102428 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c99ad0-dpflh"] Nov 04 02:59:52.585763 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:52.585724 4784 patch_prober.go:29] interesting pod/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 04 02:59:52.585935 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:52.585783 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 04 02:59:53.165247 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:53.165183 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02aa40cb-69e4-4eda-940b-180def10ab94" path="/var/lib/kubelet/pods/02aa40cb-69e4-4eda-940b-180def10ab94/volumes" Nov 04 02:59:53.590713 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:53.590621 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:59:53.591030 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:53.591002 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:59:53.591630 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:53.591613 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 02:59:54.056508 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:54.056475 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49"] Nov 04 02:59:54.062432 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:54.062399 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ac13b9-p6m49"] Nov 04 02:59:55.163265 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:55.163232 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7856e4f-cbd1-44b2-96f7-b76e64d778c5" path="/var/lib/kubelet/pods/f7856e4f-cbd1-44b2-96f7-b76e64d778c5/volumes" Nov 04 02:59:59.049737 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:59.049704 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb"] Nov 04 02:59:59.069645 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:59.069608 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f493d43d-xn4hb"] Nov 04 02:59:59.163081 ip-10-0-133-209 kubenswrapper[4784]: I1104 02:59:59.163039 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8280623a-b1b0-473f-9ef7-7b13c9ce2c7a" path="/var/lib/kubelet/pods/8280623a-b1b0-473f-9ef7-7b13c9ce2c7a/volumes" Nov 04 03:00:00.179116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.179074 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:00.187283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.187248 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.188872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.188850 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:00.193285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.193259 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.195899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.195867 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:00.199661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.199637 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.224298 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.224269 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:00.227843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.227785 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.231593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.231564 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:00.236094 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.236069 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.238150 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.238126 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:00.242797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.242775 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.244055 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.244032 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:00:00.245446 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.245430 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:00.248588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.248571 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.255816 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.255785 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkt9\" (UniqueName: \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.255935 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.255842 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.256062 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256038 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pvq\" (UniqueName: \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.256296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256276 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.256376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256354 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9762\" (UniqueName: \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.256438 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256423 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.256494 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256451 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg76b\" (UniqueName: \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.256656 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.256637 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.269401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.269378 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:00:00.269401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.269381 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:00:00.276714 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.276691 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:00.283898 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.283871 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:00.289692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.289663 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:00.332504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.332285 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:00.340667 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.340633 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:00.348483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.348454 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:00.357722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357683 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pvq\" (UniqueName: \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.357897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357812 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.357897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357847 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9762\" (UniqueName: \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.358032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357901 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.358032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357924 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg76b\" (UniqueName: \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.358032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.357957 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrxc\" (UniqueName: \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.358032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358000 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.358278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358043 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6nm\" (UniqueName: \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.358278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358087 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.358278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358134 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.358278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358238 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkt9\" (UniqueName: \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.358493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358278 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.358493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358307 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.358493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358365 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fsvh\" (UniqueName: \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.358760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358739 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.358760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.358770 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.359672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.359648 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.359783 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.359684 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.384262 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.384223 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:00.423907 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.423430 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pvq\" (UniqueName: \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.424584 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.424549 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkt9\" (UniqueName: \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.425524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.425502 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9762\" (UniqueName: \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\") pod \"maas-api-key-cleanup-29896020-478q4\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.425616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.425542 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg76b\" (UniqueName: \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.459751 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.459669 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsvh\" (UniqueName: \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.459913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.459901 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrxc\" (UniqueName: \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.459958 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.459930 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.460006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.459961 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6nm\" (UniqueName: \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.460066 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.460005 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.460118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.460101 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.460639 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.460612 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.460741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.460687 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.460741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.460714 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.503742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.503708 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:00.512470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.512445 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:00.519981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.519952 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:00.522993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.522957 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrxc\" (UniqueName: \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\") pod \"maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.523157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.523137 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6nm\" (UniqueName: \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\") pod \"maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.523268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.523145 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsvh\" (UniqueName: \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.548377 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.548067 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:00.560461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.560414 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:00.570756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.568873 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:00.577301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.574573 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:00.910686 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:00.910656 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a89c764_c3b5_4086_898f_f0a02b8e23ef.slice/crio-8e74a794c0afaf7fdd707ed4aba34dd67f00b2f196daf2890ce52269cfb03232 WatchSource:0}: Error finding container 8e74a794c0afaf7fdd707ed4aba34dd67f00b2f196daf2890ce52269cfb03232: Status 404 returned error can't find the container with id 8e74a794c0afaf7fdd707ed4aba34dd67f00b2f196daf2890ce52269cfb03232 Nov 04 03:00:00.912694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:00.912668 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:01.054945 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.054856 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z"] Nov 04 03:00:01.060640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.060607 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc8f674-72n8z"] Nov 04 03:00:01.078617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.078586 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:01.084162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.084134 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:01.131227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.130311 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:01.149558 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.149526 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:01.159445 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.159408 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce3d8f_ccb9_4260_9e43_c6fee7fdb698.slice/crio-2235ee647ed908f6e69574574251a9af6bd3b3814596b05a3419cc1654783e19 WatchSource:0}: Error finding container 2235ee647ed908f6e69574574251a9af6bd3b3814596b05a3419cc1654783e19: Status 404 returned error can't find the container with id 2235ee647ed908f6e69574574251a9af6bd3b3814596b05a3419cc1654783e19 Nov 04 03:00:01.160026 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.159996 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae8ab42_59ea_4fef_a6f4_faf8e2ff258b.slice/crio-db8b2fc6a2eb3b19482f315216ba0a6e773680f243ea7fa02ea3213c7bb2762b WatchSource:0}: Error finding container db8b2fc6a2eb3b19482f315216ba0a6e773680f243ea7fa02ea3213c7bb2762b: Status 404 returned error can't find the container with id db8b2fc6a2eb3b19482f315216ba0a6e773680f243ea7fa02ea3213c7bb2762b Nov 04 03:00:01.161401 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.161377 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5eb4f14_4660_4923_b71b_318b8e1fdbaf.slice/crio-add33f5173c7a1c96ca10e38d885738ee799fa00cff8f2f87a0d9c490e277035 WatchSource:0}: Error finding container add33f5173c7a1c96ca10e38d885738ee799fa00cff8f2f87a0d9c490e277035: Status 404 returned error can't find the container with id add33f5173c7a1c96ca10e38d885738ee799fa00cff8f2f87a0d9c490e277035 Nov 04 03:00:01.163178 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.163154 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a3ae73_f82a_407e_9a8e_9e3db46ee0b6.slice/crio-17edb02aa509e224e0c460649096a32f04c4c179a3c37e4680641b4db933e917 WatchSource:0}: Error finding container 17edb02aa509e224e0c460649096a32f04c4c179a3c37e4680641b4db933e917: Status 404 returned error can't find the container with id 17edb02aa509e224e0c460649096a32f04c4c179a3c37e4680641b4db933e917 Nov 04 03:00:01.166708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.166681 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c103f880-81b9-4174-81cb-30f8cac0245e" path="/var/lib/kubelet/pods/c103f880-81b9-4174-81cb-30f8cac0245e/volumes" Nov 04 03:00:01.228793 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.228760 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1f82d3_a565_4cad_831c_fb08f3c288ec.slice/crio-e0f345ba454233d7a3ae439e8eedbbc14269f0b298e345347de7b0719ae42b49 WatchSource:0}: Error finding container e0f345ba454233d7a3ae439e8eedbbc14269f0b298e345347de7b0719ae42b49: Status 404 returned error can't find the container with id e0f345ba454233d7a3ae439e8eedbbc14269f0b298e345347de7b0719ae42b49 Nov 04 03:00:01.230843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.230823 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:01.266519 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.266488 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:01.372654 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:01.372614 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc133df67_edc1_430e_b578_d3ec68bf5fed.slice/crio-4c8d5de507808e867a0b252c905092ae7ce53a4e7049ffb903f4759e5af8847b WatchSource:0}: Error finding container 4c8d5de507808e867a0b252c905092ae7ce53a4e7049ffb903f4759e5af8847b: Status 404 returned error can't find the container with id 4c8d5de507808e867a0b252c905092ae7ce53a4e7049ffb903f4759e5af8847b Nov 04 03:00:01.653037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.652935 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerStarted","Data":"c31e55f0ed81f9e4c28f5627867bf53731c0626a9d31aec1067800d96041ce71"} Nov 04 03:00:01.653037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.652990 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerStarted","Data":"8e74a794c0afaf7fdd707ed4aba34dd67f00b2f196daf2890ce52269cfb03232"} Nov 04 03:00:01.654642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.654611 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerStarted","Data":"4c8d5de507808e867a0b252c905092ae7ce53a4e7049ffb903f4759e5af8847b"} Nov 04 03:00:01.656173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.656099 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerStarted","Data":"db8b2fc6a2eb3b19482f315216ba0a6e773680f243ea7fa02ea3213c7bb2762b"} Nov 04 03:00:01.657764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.657738 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerStarted","Data":"17edb02aa509e224e0c460649096a32f04c4c179a3c37e4680641b4db933e917"} Nov 04 03:00:01.659496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.659468 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerStarted","Data":"add33f5173c7a1c96ca10e38d885738ee799fa00cff8f2f87a0d9c490e277035"} Nov 04 03:00:01.661068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.661044 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerStarted","Data":"e0f345ba454233d7a3ae439e8eedbbc14269f0b298e345347de7b0719ae42b49"} Nov 04 03:00:01.662827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.662799 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerStarted","Data":"2235ee647ed908f6e69574574251a9af6bd3b3814596b05a3419cc1654783e19"} Nov 04 03:00:01.772619 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:01.772562 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" podStartSLOduration=1.772542617 podStartE2EDuration="1.772542617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:01.77063699 +0000 UTC m=+1648.737137983" watchObservedRunningTime="2026-11-04 03:00:01.772542617 +0000 UTC m=+1648.739043610" Nov 04 03:00:02.683911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.683867 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerStarted","Data":"dbf85b9930a393beaee5e88c798a0efe4e3e5ae9be875abfb42030cfaf64da16"} Nov 04 03:00:02.685853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.685824 4784 generic.go:356] "Generic (PLEG): container finished" podID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerID="2ca6183e7128894e912a1c35bb3bb7779ce37f3fe1f51031d2e4e2a5e3bf1fdb" exitCode=60 Nov 04 03:00:02.686008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.685902 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerDied","Data":"2ca6183e7128894e912a1c35bb3bb7779ce37f3fe1f51031d2e4e2a5e3bf1fdb"} Nov 04 03:00:02.686162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.686144 4784 scope.go:122] "RemoveContainer" containerID="2ca6183e7128894e912a1c35bb3bb7779ce37f3fe1f51031d2e4e2a5e3bf1fdb" Nov 04 03:00:02.687750 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.687726 4784 generic.go:356] "Generic (PLEG): container finished" podID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerID="e2af41a4850f486b74b6e6ce43ffd6452995a9977c3d61d5580a5684f5e470b7" exitCode=60 Nov 04 03:00:02.687841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.687803 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerDied","Data":"e2af41a4850f486b74b6e6ce43ffd6452995a9977c3d61d5580a5684f5e470b7"} Nov 04 03:00:02.687989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.687975 4784 scope.go:122] "RemoveContainer" containerID="e2af41a4850f486b74b6e6ce43ffd6452995a9977c3d61d5580a5684f5e470b7" Nov 04 03:00:02.699055 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.698759 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerID="650146f395d900e9674b81c86094bee9e0a38a182f759ae89037ebf7a89d3312" exitCode=60 Nov 04 03:00:02.699055 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.698913 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerDied","Data":"650146f395d900e9674b81c86094bee9e0a38a182f759ae89037ebf7a89d3312"} Nov 04 03:00:02.703386 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.703082 4784 scope.go:122] "RemoveContainer" containerID="650146f395d900e9674b81c86094bee9e0a38a182f759ae89037ebf7a89d3312" Nov 04 03:00:02.707575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.707480 4784 generic.go:356] "Generic (PLEG): container finished" podID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerID="f7a34475e2af4c7475d566ea222b1a861eb381f7d1349f90ea81f62283b7a80c" exitCode=60 Nov 04 03:00:02.709948 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.707706 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerDied","Data":"f7a34475e2af4c7475d566ea222b1a861eb381f7d1349f90ea81f62283b7a80c"} Nov 04 03:00:02.709948 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.708315 4784 scope.go:122] "RemoveContainer" containerID="f7a34475e2af4c7475d566ea222b1a861eb381f7d1349f90ea81f62283b7a80c" Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.712440 4784 generic.go:356] "Generic (PLEG): container finished" podID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerID="2f03cc575d475d829f48ce5e5d028e93c945570571214ec6249a3f50fd60a259" exitCode=60 Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.712592 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerDied","Data":"2f03cc575d475d829f48ce5e5d028e93c945570571214ec6249a3f50fd60a259"} Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.713308 4784 scope.go:122] "RemoveContainer" containerID="2f03cc575d475d829f48ce5e5d028e93c945570571214ec6249a3f50fd60a259" Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.720623 4784 generic.go:356] "Generic (PLEG): container finished" podID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerID="c31e55f0ed81f9e4c28f5627867bf53731c0626a9d31aec1067800d96041ce71" exitCode=60 Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.723286 4784 scope.go:122] "RemoveContainer" containerID="c31e55f0ed81f9e4c28f5627867bf53731c0626a9d31aec1067800d96041ce71" Nov 04 03:00:02.726506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.723835 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerDied","Data":"c31e55f0ed81f9e4c28f5627867bf53731c0626a9d31aec1067800d96041ce71"} Nov 04 03:00:02.809642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:02.809581 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" podStartSLOduration=2.809562819 podStartE2EDuration="2.809562819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:02.802471736 +0000 UTC m=+1649.768972730" watchObservedRunningTime="2026-11-04 03:00:02.809562819 +0000 UTC m=+1649.776063812" Nov 04 03:00:03.761299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.761267 4784 generic.go:356] "Generic (PLEG): container finished" podID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerID="dbf85b9930a393beaee5e88c798a0efe4e3e5ae9be875abfb42030cfaf64da16" exitCode=60 Nov 04 03:00:03.761723 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.761348 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerDied","Data":"dbf85b9930a393beaee5e88c798a0efe4e3e5ae9be875abfb42030cfaf64da16"} Nov 04 03:00:03.761723 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.761687 4784 scope.go:122] "RemoveContainer" containerID="dbf85b9930a393beaee5e88c798a0efe4e3e5ae9be875abfb42030cfaf64da16" Nov 04 03:00:03.763603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.763570 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerStarted","Data":"9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd"} Nov 04 03:00:03.765894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.765850 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerStarted","Data":"bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273"} Nov 04 03:00:03.768058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.768035 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerStarted","Data":"3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d"} Nov 04 03:00:03.770243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.770217 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerStarted","Data":"670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202"} Nov 04 03:00:03.772461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.772439 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerStarted","Data":"a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875"} Nov 04 03:00:03.774534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:03.774514 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerStarted","Data":"6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd"} Nov 04 03:00:04.001314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.001250 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" podStartSLOduration=4.001234804 podStartE2EDuration="4.001234804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:03.995689307 +0000 UTC m=+1650.962190298" watchObservedRunningTime="2026-11-04 03:00:04.001234804 +0000 UTC m=+1650.967735809" Nov 04 03:00:04.034965 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.034879 4784 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce3d8f_ccb9_4260_9e43_c6fee7fdb698.slice/crio-conmon-a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce3d8f_ccb9_4260_9e43_c6fee7fdb698.slice/crio-a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875.scope\": RecentStats: unable to find data in memory cache]" Nov 04 03:00:04.118731 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.118674 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" podStartSLOduration=4.118659717 podStartE2EDuration="4.118659717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:04.114326272 +0000 UTC m=+1651.080827276" watchObservedRunningTime="2026-11-04 03:00:04.118659717 +0000 UTC m=+1651.085160708" Nov 04 03:00:04.359143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.359075 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" podStartSLOduration=4.359056664 podStartE2EDuration="4.359056664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:04.350913862 +0000 UTC m=+1651.317414894" watchObservedRunningTime="2026-11-04 03:00:04.359056664 +0000 UTC m=+1651.325557658" Nov 04 03:00:04.473527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.473473 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" podStartSLOduration=4.473457392 podStartE2EDuration="4.473457392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:04.468090381 +0000 UTC m=+1651.434591386" watchObservedRunningTime="2026-11-04 03:00:04.473457392 +0000 UTC m=+1651.439958417" Nov 04 03:00:04.586835 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.586725 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" podStartSLOduration=4.586712941 podStartE2EDuration="4.586712941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:04.58514245 +0000 UTC m=+1651.551643441" watchObservedRunningTime="2026-11-04 03:00:04.586712941 +0000 UTC m=+1651.553213933" Nov 04 03:00:04.787447 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.787409 4784 generic.go:356] "Generic (PLEG): container finished" podID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerID="a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875" exitCode=60 Nov 04 03:00:04.787862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.787500 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerDied","Data":"a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875"} Nov 04 03:00:04.787862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.787555 4784 scope.go:122] "RemoveContainer" containerID="2f03cc575d475d829f48ce5e5d028e93c945570571214ec6249a3f50fd60a259" Nov 04 03:00:04.787862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.787846 4784 scope.go:122] "RemoveContainer" containerID="a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875" Nov 04 03:00:04.788278 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.788258 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq_odh-ai-gateway-infra(27ce3d8f-ccb9-4260-9e43-c6fee7fdb698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" Nov 04 03:00:04.790352 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.790332 4784 generic.go:356] "Generic (PLEG): container finished" podID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerID="6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd" exitCode=60 Nov 04 03:00:04.790460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.790384 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerDied","Data":"6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd"} Nov 04 03:00:04.790703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.790690 4784 scope.go:122] "RemoveContainer" containerID="6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd" Nov 04 03:00:04.791705 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.791534 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb_odh-ai-gateway-infra(4a89c764-c3b5-4086-898f-f0a02b8e23ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" Nov 04 03:00:04.805437 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.805398 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerStarted","Data":"fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee"} Nov 04 03:00:04.807802 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.807772 4784 generic.go:356] "Generic (PLEG): container finished" podID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerID="9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd" exitCode=60 Nov 04 03:00:04.807931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.807864 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerDied","Data":"9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd"} Nov 04 03:00:04.808157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.808139 4784 scope.go:122] "RemoveContainer" containerID="9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd" Nov 04 03:00:04.808518 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.808490 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg_odh-ai-gateway-infra(4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" Nov 04 03:00:04.810181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.810159 4784 generic.go:356] "Generic (PLEG): container finished" podID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerID="bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273" exitCode=60 Nov 04 03:00:04.810294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.810214 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerDied","Data":"bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273"} Nov 04 03:00:04.810503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.810488 4784 scope.go:122] "RemoveContainer" containerID="bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273" Nov 04 03:00:04.810795 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.810773 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v_odh-ai-gateway-infra(95a3ae73-f82a-407e-9a8e-9e3db46ee0b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" Nov 04 03:00:04.812385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.812365 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerID="3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d" exitCode=60 Nov 04 03:00:04.812485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.812445 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerDied","Data":"3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d"} Nov 04 03:00:04.812712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.812695 4784 scope.go:122] "RemoveContainer" containerID="3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d" Nov 04 03:00:04.813040 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.813016 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs_odh-ai-gateway-infra(a5eb4f14-4660-4923-b71b-318b8e1fdbaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" Nov 04 03:00:04.814346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.814325 4784 generic.go:356] "Generic (PLEG): container finished" podID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerID="670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202" exitCode=60 Nov 04 03:00:04.814446 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.814402 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerDied","Data":"670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202"} Nov 04 03:00:04.814638 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.814621 4784 scope.go:122] "RemoveContainer" containerID="670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202" Nov 04 03:00:04.814895 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:04.814875 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896020-478q4_odh-ai-gateway-infra(6f1f82d3-a565-4cad-831c-fb08f3c288ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" Nov 04 03:00:04.816024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.816010 4784 scope.go:122] "RemoveContainer" containerID="c31e55f0ed81f9e4c28f5627867bf53731c0626a9d31aec1067800d96041ce71" Nov 04 03:00:04.831995 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.831971 4784 scope.go:122] "RemoveContainer" containerID="2ca6183e7128894e912a1c35bb3bb7779ce37f3fe1f51031d2e4e2a5e3bf1fdb" Nov 04 03:00:04.847774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.847747 4784 scope.go:122] "RemoveContainer" containerID="e2af41a4850f486b74b6e6ce43ffd6452995a9977c3d61d5580a5684f5e470b7" Nov 04 03:00:04.863789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.863764 4784 scope.go:122] "RemoveContainer" containerID="650146f395d900e9674b81c86094bee9e0a38a182f759ae89037ebf7a89d3312" Nov 04 03:00:04.879504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:04.879484 4784 scope.go:122] "RemoveContainer" containerID="f7a34475e2af4c7475d566ea222b1a861eb381f7d1349f90ea81f62283b7a80c" Nov 04 03:00:05.826576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.826547 4784 scope.go:122] "RemoveContainer" containerID="a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875" Nov 04 03:00:05.827119 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.827013 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq_odh-ai-gateway-infra(27ce3d8f-ccb9-4260-9e43-c6fee7fdb698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" Nov 04 03:00:05.828556 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.828537 4784 scope.go:122] "RemoveContainer" containerID="6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd" Nov 04 03:00:05.828911 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.828893 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb_odh-ai-gateway-infra(4a89c764-c3b5-4086-898f-f0a02b8e23ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" Nov 04 03:00:05.830200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.830163 4784 generic.go:356] "Generic (PLEG): container finished" podID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerID="fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee" exitCode=60 Nov 04 03:00:05.830268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.830241 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerDied","Data":"fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee"} Nov 04 03:00:05.830324 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.830289 4784 scope.go:122] "RemoveContainer" containerID="dbf85b9930a393beaee5e88c798a0efe4e3e5ae9be875abfb42030cfaf64da16" Nov 04 03:00:05.830535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.830514 4784 scope.go:122] "RemoveContainer" containerID="fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee" Nov 04 03:00:05.830904 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.830876 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp_odh-ai-gateway-infra(c133df67-edc1-430e-b578-d3ec68bf5fed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" Nov 04 03:00:05.832363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.832342 4784 scope.go:122] "RemoveContainer" containerID="9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd" Nov 04 03:00:05.832703 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.832681 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg_odh-ai-gateway-infra(4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" Nov 04 03:00:05.834070 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.834053 4784 scope.go:122] "RemoveContainer" containerID="bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273" Nov 04 03:00:05.834386 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.834361 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v_odh-ai-gateway-infra(95a3ae73-f82a-407e-9a8e-9e3db46ee0b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" Nov 04 03:00:05.836124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.836106 4784 scope.go:122] "RemoveContainer" containerID="3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d" Nov 04 03:00:05.836447 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.836426 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs_odh-ai-gateway-infra(a5eb4f14-4660-4923-b71b-318b8e1fdbaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" Nov 04 03:00:05.837700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:05.837684 4784 scope.go:122] "RemoveContainer" containerID="670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202" Nov 04 03:00:05.837939 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:05.837922 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896020-478q4_odh-ai-gateway-infra(6f1f82d3-a565-4cad-831c-fb08f3c288ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" Nov 04 03:00:06.402418 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:06.402387 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:00:06.935274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:06.933802 4784 scope.go:122] "RemoveContainer" containerID="fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee" Nov 04 03:00:06.942253 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:06.941179 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp_odh-ai-gateway-infra(c133df67-edc1-430e-b578-d3ec68bf5fed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" Nov 04 03:00:15.154144 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:15.154046 4784 scope.go:122] "RemoveContainer" containerID="3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d" Nov 04 03:00:15.154640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:15.154416 4784 scope.go:122] "RemoveContainer" containerID="6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd" Nov 04 03:00:16.207894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:16.207856 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerStarted","Data":"80b800cdb48cbd47f887899f71232bb1699727164a5ff9533ade611028d02d0e"} Nov 04 03:00:16.210266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:16.210236 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerStarted","Data":"27b2660d0a7296ddc844c55636166185c4208f97cd359448c6266dd52c46d2a7"} Nov 04 03:00:17.050453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.050418 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4"] Nov 04 03:00:17.057232 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.057182 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45fd7b5-jvnk4"] Nov 04 03:00:17.153869 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.153833 4784 scope.go:122] "RemoveContainer" containerID="670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202" Nov 04 03:00:17.154324 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.154300 4784 scope.go:122] "RemoveContainer" containerID="9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd" Nov 04 03:00:17.171975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.171943 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73054940-7c23-43aa-b67f-8baa65c9a2f8" path="/var/lib/kubelet/pods/73054940-7c23-43aa-b67f-8baa65c9a2f8/volumes" Nov 04 03:00:17.222806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.222774 4784 generic.go:356] "Generic (PLEG): container finished" podID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerID="80b800cdb48cbd47f887899f71232bb1699727164a5ff9533ade611028d02d0e" exitCode=60 Nov 04 03:00:17.223279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.222848 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerDied","Data":"80b800cdb48cbd47f887899f71232bb1699727164a5ff9533ade611028d02d0e"} Nov 04 03:00:17.223279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.222915 4784 scope.go:122] "RemoveContainer" containerID="6f77b8e0ab9fa4a44927c982c13174b0cd3a0d6f9a1434c557ab912dcc3d97cd" Nov 04 03:00:17.223485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.223462 4784 scope.go:122] "RemoveContainer" containerID="80b800cdb48cbd47f887899f71232bb1699727164a5ff9533ade611028d02d0e" Nov 04 03:00:17.223942 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:17.223922 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb_odh-ai-gateway-infra(4a89c764-c3b5-4086-898f-f0a02b8e23ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" Nov 04 03:00:17.225124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.225104 4784 generic.go:356] "Generic (PLEG): container finished" podID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerID="27b2660d0a7296ddc844c55636166185c4208f97cd359448c6266dd52c46d2a7" exitCode=60 Nov 04 03:00:17.225221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.225170 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerDied","Data":"27b2660d0a7296ddc844c55636166185c4208f97cd359448c6266dd52c46d2a7"} Nov 04 03:00:17.225585 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.225570 4784 scope.go:122] "RemoveContainer" containerID="27b2660d0a7296ddc844c55636166185c4208f97cd359448c6266dd52c46d2a7" Nov 04 03:00:17.225977 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:17.225954 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs_odh-ai-gateway-infra(a5eb4f14-4660-4923-b71b-318b8e1fdbaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" Nov 04 03:00:17.241253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.241231 4784 scope.go:122] "RemoveContainer" containerID="3306592bbe49bb5dd81500277a49fa6e37848e77034946490a17e0ca400f713d" Nov 04 03:00:17.340138 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.340046 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:17.470961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:17.469137 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:18.052342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.052305 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm"] Nov 04 03:00:18.058124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.058091 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w"] Nov 04 03:00:18.064493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.064448 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015f46a4-7flpm"] Nov 04 03:00:18.070554 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.070524 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1b5482-r8g9w"] Nov 04 03:00:18.153904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.153876 4784 scope.go:122] "RemoveContainer" containerID="a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875" Nov 04 03:00:18.239149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.239118 4784 generic.go:356] "Generic (PLEG): container finished" podID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerID="e115858dceb477d0cbe2734a173cc0c40fc6bf94f115c5ab2b9d1014d29cb52c" exitCode=60 Nov 04 03:00:18.239149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.239140 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerDied","Data":"e115858dceb477d0cbe2734a173cc0c40fc6bf94f115c5ab2b9d1014d29cb52c"} Nov 04 03:00:18.239627 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.239239 4784 scope.go:122] "RemoveContainer" containerID="670b1bd51e41f373f048c113071cf1258a1fc062546e3b810e795cb3b9fd6202" Nov 04 03:00:18.239754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.239736 4784 scope.go:122] "RemoveContainer" containerID="e115858dceb477d0cbe2734a173cc0c40fc6bf94f115c5ab2b9d1014d29cb52c" Nov 04 03:00:18.240353 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:18.240218 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896020-478q4_odh-ai-gateway-infra(6f1f82d3-a565-4cad-831c-fb08f3c288ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" Nov 04 03:00:18.244233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.244206 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerStarted","Data":"cd081cfcf993299abf8e1bea5bcd35acca05e904de777cf799711a030af86454"} Nov 04 03:00:18.531643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.531592 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:18.540873 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.540840 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:18.581495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.581460 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\") pod \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " Nov 04 03:00:18.582036 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.582015 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\") pod \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " Nov 04 03:00:18.582387 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.582318 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca" pod "a5eb4f14-4660-4923-b71b-318b8e1fdbaf" (UID: "a5eb4f14-4660-4923-b71b-318b8e1fdbaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:18.582525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.582401 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca" pod "4a89c764-c3b5-4086-898f-f0a02b8e23ef" (UID: "4a89c764-c3b5-4086-898f-f0a02b8e23ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:18.582693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.582672 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\" (UniqueName: \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\") pod \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\" (UID: \"4a89c764-c3b5-4086-898f-f0a02b8e23ef\") " Nov 04 03:00:18.582929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.582913 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\" (UniqueName: \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\") pod \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\" (UID: \"a5eb4f14-4660-4923-b71b-318b8e1fdbaf\") " Nov 04 03:00:18.584288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.584162 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:18.584462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.584446 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a89c764-c3b5-4086-898f-f0a02b8e23ef-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:18.585706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.585662 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq" pod "4a89c764-c3b5-4086-898f-f0a02b8e23ef" (UID: "4a89c764-c3b5-4086-898f-f0a02b8e23ef"). InnerVolumeSpecName "kube-api-access-44pvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:18.586705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.586685 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9" pod "a5eb4f14-4660-4923-b71b-318b8e1fdbaf" (UID: "a5eb4f14-4660-4923-b71b-318b8e1fdbaf"). InnerVolumeSpecName "kube-api-access-nfkt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:18.686382 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.686341 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkt9\" (UniqueName: \"kubernetes.io/projected/a5eb4f14-4660-4923-b71b-318b8e1fdbaf-kube-api-access-nfkt9\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:18.686382 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:18.686383 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pvq\" (UniqueName: \"kubernetes.io/projected/4a89c764-c3b5-4086-898f-f0a02b8e23ef-kube-api-access-44pvq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:19.051284 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.051251 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq"] Nov 04 03:00:19.057112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.057083 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59d253f-8kzgq"] Nov 04 03:00:19.153619 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.153585 4784 scope.go:122] "RemoveContainer" containerID="bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273" Nov 04 03:00:19.153935 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.153911 4784 scope.go:122] "RemoveContainer" containerID="fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee" Nov 04 03:00:19.166059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.166029 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a3a56e-1335-450d-8288-6ceefa517109" path="/var/lib/kubelet/pods/51a3a56e-1335-450d-8288-6ceefa517109/volumes" Nov 04 03:00:19.166611 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.166589 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d352ef6-2d6e-4854-b9a8-19fd2686f641" path="/var/lib/kubelet/pods/5d352ef6-2d6e-4854-b9a8-19fd2686f641/volumes" Nov 04 03:00:19.167091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.167071 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c084feb0-7589-49da-858a-b750a9424bb9" path="/var/lib/kubelet/pods/c084feb0-7589-49da-858a-b750a9424bb9/volumes" Nov 04 03:00:19.257880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.257847 4784 generic.go:356] "Generic (PLEG): container finished" podID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerID="cd081cfcf993299abf8e1bea5bcd35acca05e904de777cf799711a030af86454" exitCode=60 Nov 04 03:00:19.258345 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.257887 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerDied","Data":"cd081cfcf993299abf8e1bea5bcd35acca05e904de777cf799711a030af86454"} Nov 04 03:00:19.258345 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.257960 4784 scope.go:122] "RemoveContainer" containerID="9b3429fed32b0e993a0dc96f3d9d357a9a85da67575c163f59e0715ab7b471dd" Nov 04 03:00:19.258592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.258572 4784 scope.go:122] "RemoveContainer" containerID="cd081cfcf993299abf8e1bea5bcd35acca05e904de777cf799711a030af86454" Nov 04 03:00:19.259090 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:19.259062 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg_odh-ai-gateway-infra(4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" Nov 04 03:00:19.261453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.261428 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" event={"ID":"a5eb4f14-4660-4923-b71b-318b8e1fdbaf","Type":"ContainerDied","Data":"add33f5173c7a1c96ca10e38d885738ee799fa00cff8f2f87a0d9c490e277035"} Nov 04 03:00:19.261555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.261435 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs" Nov 04 03:00:19.267247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.267223 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerStarted","Data":"fdb311676077db665b6dd87aaaee6840329ba1654bfba59eaa382eae1e2c1630"} Nov 04 03:00:19.269487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.269467 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" Nov 04 03:00:19.269610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.269483 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb" event={"ID":"4a89c764-c3b5-4086-898f-f0a02b8e23ef","Type":"ContainerDied","Data":"8e74a794c0afaf7fdd707ed4aba34dd67f00b2f196daf2890ce52269cfb03232"} Nov 04 03:00:19.280611 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.280590 4784 scope.go:122] "RemoveContainer" containerID="27b2660d0a7296ddc844c55636166185c4208f97cd359448c6266dd52c46d2a7" Nov 04 03:00:19.291943 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.291921 4784 scope.go:122] "RemoveContainer" containerID="80b800cdb48cbd47f887899f71232bb1699727164a5ff9533ade611028d02d0e" Nov 04 03:00:19.491672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.491639 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:19.498142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.498113 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:19.504807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.504776 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe4e0c-29896020-56vrs"] Nov 04 03:00:19.672250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.663892 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:19.867960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.867872 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cdcf67-29896020-lhfrb"] Nov 04 03:00:19.876738 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.876711 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:19.990694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:19.990670 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:20.063545 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.063514 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\" (UniqueName: \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\") pod \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " Nov 04 03:00:20.063845 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.063830 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\") pod \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\" (UID: \"6f1f82d3-a565-4cad-831c-fb08f3c288ec\") " Nov 04 03:00:20.064278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.064254 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca" pod "6f1f82d3-a565-4cad-831c-fb08f3c288ec" (UID: "6f1f82d3-a565-4cad-831c-fb08f3c288ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:20.064518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.064505 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1f82d3-a565-4cad-831c-fb08f3c288ec-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:20.065689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.065669 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762" pod "6f1f82d3-a565-4cad-831c-fb08f3c288ec" (UID: "6f1f82d3-a565-4cad-831c-fb08f3c288ec"). InnerVolumeSpecName "kube-api-access-n9762". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:20.165456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.165424 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9762\" (UniqueName: \"kubernetes.io/projected/6f1f82d3-a565-4cad-831c-fb08f3c288ec-kube-api-access-n9762\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:20.283309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.283277 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerStarted","Data":"efb49ec655395446ab8778c3367b9080d955cc403a7b8ab234e166c98d988d68"} Nov 04 03:00:20.287147 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.287116 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerStarted","Data":"42effe9981eba5ed928d8358f83636b854e0415c256b58b520d6da796d0fa7a9"} Nov 04 03:00:20.289466 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.289448 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" Nov 04 03:00:20.289571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.289477 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4" event={"ID":"6f1f82d3-a565-4cad-831c-fb08f3c288ec","Type":"ContainerDied","Data":"e0f345ba454233d7a3ae439e8eedbbc14269f0b298e345347de7b0719ae42b49"} Nov 04 03:00:20.289571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.289530 4784 scope.go:122] "RemoveContainer" containerID="e115858dceb477d0cbe2734a173cc0c40fc6bf94f115c5ab2b9d1014d29cb52c" Nov 04 03:00:20.291887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.291862 4784 generic.go:356] "Generic (PLEG): container finished" podID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerID="fdb311676077db665b6dd87aaaee6840329ba1654bfba59eaa382eae1e2c1630" exitCode=60 Nov 04 03:00:20.291977 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.291908 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerDied","Data":"fdb311676077db665b6dd87aaaee6840329ba1654bfba59eaa382eae1e2c1630"} Nov 04 03:00:20.292370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.292353 4784 scope.go:122] "RemoveContainer" containerID="fdb311676077db665b6dd87aaaee6840329ba1654bfba59eaa382eae1e2c1630" Nov 04 03:00:20.292699 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:20.292679 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq_odh-ai-gateway-infra(27ce3d8f-ccb9-4260-9e43-c6fee7fdb698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" Nov 04 03:00:20.304515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.304461 4784 scope.go:122] "RemoveContainer" containerID="a39c65aa0002f683b435a7c63c415287d2dd0c63a236f54d7b098f1549389875" Nov 04 03:00:20.520825 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.520788 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:20.528403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.528377 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896020-478q4"] Nov 04 03:00:20.618346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.618318 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:20.672721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.672683 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\" (UniqueName: \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\") pod \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " Nov 04 03:00:20.673043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.673027 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\") pod \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\" (UID: \"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b\") " Nov 04 03:00:20.673436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.673413 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca" pod "4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" (UID: "4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:20.673749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.673732 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:20.674974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.674943 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm" pod "4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" (UID: "4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b"). InnerVolumeSpecName "kube-api-access-hg6nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:20.770452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.770363 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:20.775057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:20.775028 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6nm\" (UniqueName: \"kubernetes.io/projected/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b-kube-api-access-hg6nm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.165328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.165304 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" path="/var/lib/kubelet/pods/4a89c764-c3b5-4086-898f-f0a02b8e23ef/volumes" Nov 04 03:00:21.165805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.165791 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" path="/var/lib/kubelet/pods/6f1f82d3-a565-4cad-831c-fb08f3c288ec/volumes" Nov 04 03:00:21.166220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.166208 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" path="/var/lib/kubelet/pods/a5eb4f14-4660-4923-b71b-318b8e1fdbaf/volumes" Nov 04 03:00:21.308229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.308171 4784 generic.go:356] "Generic (PLEG): container finished" podID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerID="efb49ec655395446ab8778c3367b9080d955cc403a7b8ab234e166c98d988d68" exitCode=60 Nov 04 03:00:21.308229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.308215 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerDied","Data":"efb49ec655395446ab8778c3367b9080d955cc403a7b8ab234e166c98d988d68"} Nov 04 03:00:21.308781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.308279 4784 scope.go:122] "RemoveContainer" containerID="fb3a4e74520b6df7995f793b10e1d8a141da3a6b7dc3a48c5d3e53d0595929ee" Nov 04 03:00:21.309302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.309283 4784 scope.go:122] "RemoveContainer" containerID="efb49ec655395446ab8778c3367b9080d955cc403a7b8ab234e166c98d988d68" Nov 04 03:00:21.309755 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:21.309732 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp_odh-ai-gateway-infra(c133df67-edc1-430e-b578-d3ec68bf5fed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" Nov 04 03:00:21.310441 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.310418 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" Nov 04 03:00:21.310532 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.310470 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg" event={"ID":"4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b","Type":"ContainerDied","Data":"db8b2fc6a2eb3b19482f315216ba0a6e773680f243ea7fa02ea3213c7bb2762b"} Nov 04 03:00:21.313090 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.312750 4784 generic.go:356] "Generic (PLEG): container finished" podID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerID="42effe9981eba5ed928d8358f83636b854e0415c256b58b520d6da796d0fa7a9" exitCode=60 Nov 04 03:00:21.313090 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.312902 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerDied","Data":"42effe9981eba5ed928d8358f83636b854e0415c256b58b520d6da796d0fa7a9"} Nov 04 03:00:21.313639 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.313617 4784 scope.go:122] "RemoveContainer" containerID="42effe9981eba5ed928d8358f83636b854e0415c256b58b520d6da796d0fa7a9" Nov 04 03:00:21.314062 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:21.314036 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v_odh-ai-gateway-infra(95a3ae73-f82a-407e-9a8e-9e3db46ee0b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" Nov 04 03:00:21.326305 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.326283 4784 scope.go:122] "RemoveContainer" containerID="cd081cfcf993299abf8e1bea5bcd35acca05e904de777cf799711a030af86454" Nov 04 03:00:21.341500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.341478 4784 scope.go:122] "RemoveContainer" containerID="bce93aa46d78fb30e9d50962e6142741a9f6b7aed3bbec887c68c93558013273" Nov 04 03:00:21.426424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.425432 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:21.503154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.503128 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:21.587064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.587033 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\" (UniqueName: \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\") pod \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " Nov 04 03:00:21.587255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.587177 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\") pod \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\" (UID: \"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698\") " Nov 04 03:00:21.587546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.587522 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca" pod "27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" (UID: "27ce3d8f-ccb9-4260-9e43-c6fee7fdb698"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:21.587813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.587796 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.589496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.589468 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh" pod "27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" (UID: "27ce3d8f-ccb9-4260-9e43-c6fee7fdb698"). InnerVolumeSpecName "kube-api-access-7fsvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:21.653260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.653229 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:21.667104 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.667066 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:21.672913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.672875 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ebb7fd-29896020-stsjg"] Nov 04 03:00:21.689926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:21.689846 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fsvh\" (UniqueName: \"kubernetes.io/projected/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698-kube-api-access-7fsvh\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:22.344672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.344631 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" event={"ID":"27ce3d8f-ccb9-4260-9e43-c6fee7fdb698","Type":"ContainerDied","Data":"2235ee647ed908f6e69574574251a9af6bd3b3814596b05a3419cc1654783e19"} Nov 04 03:00:22.344981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.344755 4784 scope.go:122] "RemoveContainer" containerID="fdb311676077db665b6dd87aaaee6840329ba1654bfba59eaa382eae1e2c1630" Nov 04 03:00:22.344981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.344908 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq" Nov 04 03:00:22.683493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.683463 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:22.688073 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.688047 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:22.722464 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.712477 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:22.724393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.724365 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896020-lp5cq"] Nov 04 03:00:22.836399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.836335 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\") pod \"c133df67-edc1-430e-b578-d3ec68bf5fed\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " Nov 04 03:00:22.836613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.836595 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\" (UniqueName: \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\") pod \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " Nov 04 03:00:22.836692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.836622 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\" (UniqueName: \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\") pod \"c133df67-edc1-430e-b578-d3ec68bf5fed\" (UID: \"c133df67-edc1-430e-b578-d3ec68bf5fed\") " Nov 04 03:00:22.836692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.836674 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca" pod "c133df67-edc1-430e-b578-d3ec68bf5fed" (UID: "c133df67-edc1-430e-b578-d3ec68bf5fed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:22.836807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.836755 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\") pod \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\" (UID: \"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6\") " Nov 04 03:00:22.837136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.837106 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca" pod "95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" (UID: "95a3ae73-f82a-407e-9a8e-9e3db46ee0b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:22.837605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.837584 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:22.837687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.837611 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c133df67-edc1-430e-b578-d3ec68bf5fed-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:22.839297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.839272 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc" pod "c133df67-edc1-430e-b578-d3ec68bf5fed" (UID: "c133df67-edc1-430e-b578-d3ec68bf5fed"). InnerVolumeSpecName "kube-api-access-mlrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:22.839517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.839494 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b" pod "95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" (UID: "95a3ae73-f82a-407e-9a8e-9e3db46ee0b6"). InnerVolumeSpecName "kube-api-access-gg76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:23.009216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.987993 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg76b\" (UniqueName: \"kubernetes.io/projected/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6-kube-api-access-gg76b\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:23.009216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:22.988032 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrxc\" (UniqueName: \"kubernetes.io/projected/c133df67-edc1-430e-b578-d3ec68bf5fed-kube-api-access-mlrxc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:23.081215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.077395 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d"] Nov 04 03:00:23.200058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.200024 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" path="/var/lib/kubelet/pods/27ce3d8f-ccb9-4260-9e43-c6fee7fdb698/volumes" Nov 04 03:00:23.201226 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.201174 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" path="/var/lib/kubelet/pods/4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b/volumes" Nov 04 03:00:23.202562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.202524 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d"] Nov 04 03:00:23.237765 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:23.237720 4784 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-5ef3b302-j7l2d\" is forbidden: User \"system:node:ip-10-0-133-209.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-209.us-west-1.compute.internal' and this object" podUID="555c8593-a5dc-4142-bd3d-5a700673f12a" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef3b302-j7l2d" Nov 04 03:00:23.355807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.355714 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" Nov 04 03:00:23.355807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.355740 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp" event={"ID":"c133df67-edc1-430e-b578-d3ec68bf5fed","Type":"ContainerDied","Data":"4c8d5de507808e867a0b252c905092ae7ce53a4e7049ffb903f4759e5af8847b"} Nov 04 03:00:23.355807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.355804 4784 scope.go:122] "RemoveContainer" containerID="efb49ec655395446ab8778c3367b9080d955cc403a7b8ab234e166c98d988d68" Nov 04 03:00:23.357856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.357835 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" Nov 04 03:00:23.357979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.357877 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v" event={"ID":"95a3ae73-f82a-407e-9a8e-9e3db46ee0b6","Type":"ContainerDied","Data":"17edb02aa509e224e0c460649096a32f04c4c179a3c37e4680641b4db933e917"} Nov 04 03:00:23.377824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.377796 4784 scope.go:122] "RemoveContainer" containerID="42effe9981eba5ed928d8358f83636b854e0415c256b58b520d6da796d0fa7a9" Nov 04 03:00:23.586211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.577452 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:23.587153 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.587119 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b7476-29896020-f2x9v"] Nov 04 03:00:23.675268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.675237 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:23.680800 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:23.680771 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4d5a0200-29896020-bvzkp"] Nov 04 03:00:25.171233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:25.171144 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555c8593-a5dc-4142-bd3d-5a700673f12a" path="/var/lib/kubelet/pods/555c8593-a5dc-4142-bd3d-5a700673f12a/volumes" Nov 04 03:00:25.172577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:25.172552 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" path="/var/lib/kubelet/pods/95a3ae73-f82a-407e-9a8e-9e3db46ee0b6/volumes" Nov 04 03:00:25.172988 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:25.172973 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" path="/var/lib/kubelet/pods/c133df67-edc1-430e-b578-d3ec68bf5fed/volumes" Nov 04 03:00:26.670692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.670656 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng"] Nov 04 03:00:26.671470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671455 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671471 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671483 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671489 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671499 4784 cpu_manager.go:414] "containerMap: removing container" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671504 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671509 4784 cpu_manager.go:414] "containerMap: removing container" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671514 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671520 4784 cpu_manager.go:414] "containerMap: removing container" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671524 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671530 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671535 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671544 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671549 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671564 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671569 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671574 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671578 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671589 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671593 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671608 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671612 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671620 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671624 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671634 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671638 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671646 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671652 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671659 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671663 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671668 4784 cpu_manager.go:414] "containerMap: removing container" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671672 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671683 4784 cpu_manager.go:414] "containerMap: removing container" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671687 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671695 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671699 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671707 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671711 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671726 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671731 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671842 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671848 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671857 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671865 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae8ab42-59ea-4fef-a6f4-faf8e2ff258b" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671872 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671877 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671882 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671887 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671893 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671897 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671904 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671910 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671915 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671919 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671926 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671932 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:00:26.671946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.671937 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3ae73-f82a-407e-9a8e-9e3db46ee0b6" containerName="cleanup" Nov 04 03:00:26.677127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.677108 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.733580 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.733548 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng"] Nov 04 03:00:26.777160 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.777114 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:00:26.777359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.777241 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:00:26.777359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.777353 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:00:26.777504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.777415 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:00:26.825821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.825739 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.825958 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.825864 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.825958 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.825943 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jdc\" (UniqueName: \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.927077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.927036 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.927320 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.927122 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jdc\" (UniqueName: \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.927516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.927486 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.928064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.928037 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.929978 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.929955 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.989799 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.989771 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jdc\" (UniqueName: \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\") pod \"maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:26.991769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:26.991747 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:27.379129 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:00:27.379086 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe384e7_8c53_4fa6_9f91_1dc3985ad2de.slice/crio-2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0 WatchSource:0}: Error finding container 2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0: Status 404 returned error can't find the container with id 2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0 Nov 04 03:00:27.380742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:27.380716 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng"] Nov 04 03:00:27.416543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:27.416378 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" event={"ID":"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de","Type":"ContainerStarted","Data":"2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0"} Nov 04 03:00:28.428675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:28.428644 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" event={"ID":"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de","Type":"ContainerStarted","Data":"27c70b8a6600d5b24262b5c393f3345aa5d8a2225bbfcbc9daa1bd0213be66ae"} Nov 04 03:00:28.550557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:28.550500 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" podStartSLOduration=2.550484898 podStartE2EDuration="2.550484898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:28.546977555 +0000 UTC m=+1675.513478544" watchObservedRunningTime="2026-11-04 03:00:28.550484898 +0000 UTC m=+1675.516985889" Nov 04 03:00:29.056107 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:29.056071 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4"] Nov 04 03:00:29.062549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:29.062456 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8d90f-4ppx4"] Nov 04 03:00:29.199940 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:29.199884 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2961fc3-c8e4-405a-9cc2-4ebc92ece520" path="/var/lib/kubelet/pods/b2961fc3-c8e4-405a-9cc2-4ebc92ece520/volumes" Nov 04 03:00:29.440597 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:29.440565 4784 generic.go:356] "Generic (PLEG): container finished" podID="dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" containerID="27c70b8a6600d5b24262b5c393f3345aa5d8a2225bbfcbc9daa1bd0213be66ae" exitCode=0 Nov 04 03:00:29.440994 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:29.440655 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" event={"ID":"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de","Type":"ContainerDied","Data":"27c70b8a6600d5b24262b5c393f3345aa5d8a2225bbfcbc9daa1bd0213be66ae"} Nov 04 03:00:30.055414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.055383 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk"] Nov 04 03:00:30.061605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.061575 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f098c46-6stkk"] Nov 04 03:00:30.634581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.634553 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:30.713956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.713873 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\") pod \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " Nov 04 03:00:30.713956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.713919 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\") pod \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " Nov 04 03:00:30.714233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.713988 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\" (UniqueName: \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\") pod \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\" (UID: \"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de\") " Nov 04 03:00:30.714397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.714373 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca" pod "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" (UID: "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:30.714673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.714652 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:30.716328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.716286 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca" pod "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" (UID: "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:00:30.716481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.716461 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc" pod "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" (UID: "dbe384e7-8c53-4fa6-9f91-1dc3985ad2de"). InnerVolumeSpecName "kube-api-access-97jdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:30.816515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.816475 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:30.816515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:30.816505 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jdc\" (UniqueName: \"kubernetes.io/projected/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de-kube-api-access-97jdc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:31.059576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.059470 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft"] Nov 04 03:00:31.065316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.065276 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa36587c-w8sft"] Nov 04 03:00:31.164290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.164254 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ea6d41-fe3e-459c-9f87-24c138f380c7" path="/var/lib/kubelet/pods/27ea6d41-fe3e-459c-9f87-24c138f380c7/volumes" Nov 04 03:00:31.164841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.164824 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af" path="/var/lib/kubelet/pods/cbab3d1f-6ca1-4f3e-9002-b3bfde17f0af/volumes" Nov 04 03:00:31.472775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.472742 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" event={"ID":"dbe384e7-8c53-4fa6-9f91-1dc3985ad2de","Type":"ContainerDied","Data":"2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0"} Nov 04 03:00:31.472775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.472767 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng" Nov 04 03:00:31.473002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:31.472792 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb5566b9d2239cfec61d61e331f89cedb40cad607e37067bd9e022b6a9933d0" Nov 04 03:00:33.052281 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:33.052247 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k"] Nov 04 03:00:33.060799 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:33.060773 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355cbc8a-wsv5k"] Nov 04 03:00:33.168564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:33.168522 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830882a3-067d-49bf-ade4-d486a6b166eb" path="/var/lib/kubelet/pods/830882a3-067d-49bf-ade4-d486a6b166eb/volumes" Nov 04 03:00:34.143563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.143526 4784 scope.go:122] "RemoveContainer" containerID="be1904b2107de301c9e5d427fb02f54100dfb0898e6e388ba2b38cfcb28917a0" Nov 04 03:00:34.155252 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.155229 4784 scope.go:122] "RemoveContainer" containerID="41267eb7636a790047f7f5143c60c8ea9da09af01a2b788746de804c4862ee1a" Nov 04 03:00:34.166279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.166261 4784 scope.go:122] "RemoveContainer" containerID="5017c1497023922f0ae81d150d5dea927b0d9e9f005b12619f7efe73677086fd" Nov 04 03:00:34.177492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.177473 4784 scope.go:122] "RemoveContainer" containerID="469d8131cf3d00f54265dbbf205e22d319c1039b5c0ff9ac28521e3f9fa3be15" Nov 04 03:00:34.188880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.188859 4784 scope.go:122] "RemoveContainer" containerID="cf10de6cc13e8aa4edc0485045982ccd7b1704c9f20e07e5d4ef677283a45ae1" Nov 04 03:00:34.202439 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:34.202416 4784 scope.go:122] "RemoveContainer" containerID="1d2c65b77630804af34dbf1adc716b7a41eb30379f8e44d854ab9f46ddbcc652" Nov 04 03:00:36.402005 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:36.401969 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:00:36.908947 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:36.908910 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 03:00:36.909317 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:36.909262 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" containerID="cri-o://d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2" gracePeriod=30 Nov 04 03:00:36.969241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:36.963288 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 03:00:36.969241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:36.963518 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" containerID="cri-o://f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56" gracePeriod=30 Nov 04 03:00:40.744493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:40.744442 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:40.744920 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:40.744507 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:41.050903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:41.050818 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl"] Nov 04 03:00:41.108366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:41.108286 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fd5cb-4zvfl"] Nov 04 03:00:41.164126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:41.164088 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535e4550-706e-4dea-bac1-f057ae31abfa" path="/var/lib/kubelet/pods/535e4550-706e-4dea-bac1-f057ae31abfa/volumes" Nov 04 03:00:41.379411 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:41.379320 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:41.379411 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:41.379371 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:42.269063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.269029 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 03:00:42.330838 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.330813 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 03:00:42.401001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.400970 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401027 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401067 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401129 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401147 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401272 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401298 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.401468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401350 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401384 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.401687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401503 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.401687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401512 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.401687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401613 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.401855 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.401716 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\") pod \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\" (UID: \"24cb22f0-97b4-42af-8f2e-6adc4b331bdb\") " Nov 04 03:00:42.402223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402015 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:42.402867 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402839 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.402961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402870 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.402961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402885 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.402961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402899 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.402961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.402912 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.403754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.403724 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.403851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.403772 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "kube-api-access-7lfmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:42.403851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.403806 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:42.403851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.403806 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token" pod "24cb22f0-97b4-42af-8f2e-6adc4b331bdb" (UID: "24cb22f0-97b4-42af-8f2e-6adc4b331bdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:42.503781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.503677 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.503976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.503789 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504035 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504058 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.504092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504085 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504119 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504160 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504246 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.504347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504304 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.504347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504309 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504372 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\") pod \"f0f4f7bc-152f-477b-827f-2099474380bc\" (UID: \"f0f4f7bc-152f-477b-827f-2099474380bc\") " Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504504 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504529 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.504908 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505260 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505283 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505300 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505314 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4f7bc-152f-477b-827f-2099474380bc-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505329 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505343 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505357 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505370 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.506459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.505384 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfmm\" (UniqueName: \"kubernetes.io/projected/24cb22f0-97b4-42af-8f2e-6adc4b331bdb-kube-api-access-7lfmm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.507854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.507824 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "kube-api-access-pwpnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:42.508684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.508657 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:42.511380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.511352 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:42.511494 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.511431 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo" pod "f0f4f7bc-152f-477b-827f-2099474380bc" (UID: "f0f4f7bc-152f-477b-827f-2099474380bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:42.607211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.607162 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.607423 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.607217 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpnv\" (UniqueName: \"kubernetes.io/projected/f0f4f7bc-152f-477b-827f-2099474380bc-kube-api-access-pwpnv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.607423 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.607234 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f4f7bc-152f-477b-827f-2099474380bc-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.607423 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.607248 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f4f7bc-152f-477b-827f-2099474380bc-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:42.634213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.634163 4784 generic.go:356] "Generic (PLEG): container finished" podID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerID="d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2" exitCode=0 Nov 04 03:00:42.634417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.634252 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" event={"ID":"24cb22f0-97b4-42af-8f2e-6adc4b331bdb","Type":"ContainerDied","Data":"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2"} Nov 04 03:00:42.634417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.634270 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" Nov 04 03:00:42.634417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.634311 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm" event={"ID":"24cb22f0-97b4-42af-8f2e-6adc4b331bdb","Type":"ContainerDied","Data":"cdc769e51ef83c541fbd2eccd42f7538ec9159b42df4683b1ec7948650fd9cf7"} Nov 04 03:00:42.634417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.634343 4784 scope.go:122] "RemoveContainer" containerID="d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2" Nov 04 03:00:42.636328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.636299 4784 generic.go:356] "Generic (PLEG): container finished" podID="f0f4f7bc-152f-477b-827f-2099474380bc" containerID="f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56" exitCode=0 Nov 04 03:00:42.636462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.636358 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" Nov 04 03:00:42.636462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.636388 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" event={"ID":"f0f4f7bc-152f-477b-827f-2099474380bc","Type":"ContainerDied","Data":"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56"} Nov 04 03:00:42.636462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.636430 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm" event={"ID":"f0f4f7bc-152f-477b-827f-2099474380bc","Type":"ContainerDied","Data":"4baf1234326b5bf11057ded51bd85d781428e7587bd23a9d7d0908ec220b1cc5"} Nov 04 03:00:42.648302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.648131 4784 scope.go:122] "RemoveContainer" containerID="d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2" Nov 04 03:00:42.648572 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:42.648541 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2\": container with ID starting with d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2 not found: ID does not exist" containerID="d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2" Nov 04 03:00:42.648695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.648583 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2"} err="failed to get container status \"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2\": rpc error: code = NotFound desc = could not find container \"d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2\": container with ID starting with d1deff7f8b529bce96ca2bd480064b1b48c832b6d6eed2744387a410728a3eb2 not found: ID does not exist" Nov 04 03:00:42.648695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.648609 4784 scope.go:122] "RemoveContainer" containerID="f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56" Nov 04 03:00:42.660804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.660778 4784 scope.go:122] "RemoveContainer" containerID="f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56" Nov 04 03:00:42.661158 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:00:42.661139 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56\": container with ID starting with f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56 not found: ID does not exist" containerID="f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56" Nov 04 03:00:42.661266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.661169 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56"} err="failed to get container status \"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56\": rpc error: code = NotFound desc = could not find container \"f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56\": container with ID starting with f632dddf6bd53e6e561a4f488bec833384f4973ff5f466c1a57092c4f546db56 not found: ID does not exist" Nov 04 03:00:42.806660 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.806575 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 03:00:42.816609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.816579 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-v5wsm"] Nov 04 03:00:42.929602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.929573 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 03:00:42.934436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:42.934411 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-d5qtm"] Nov 04 03:00:43.163131 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:43.163047 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" path="/var/lib/kubelet/pods/24cb22f0-97b4-42af-8f2e-6adc4b331bdb/volumes" Nov 04 03:00:43.163663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:43.163646 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" path="/var/lib/kubelet/pods/f0f4f7bc-152f-477b-827f-2099474380bc/volumes" Nov 04 03:00:47.051254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:47.051216 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5"] Nov 04 03:00:47.103593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:47.103563 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af2588-5xsk5"] Nov 04 03:00:47.163411 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:47.163382 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c55449-0261-41f0-b602-ac86f3706e01" path="/var/lib/kubelet/pods/74c55449-0261-41f0-b602-ac86f3706e01/volumes" Nov 04 03:00:56.056856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:56.056818 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz"] Nov 04 03:00:56.105956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:56.105919 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77a76cd7-whsrz"] Nov 04 03:00:57.162923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:00:57.162883 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a42e57-9048-4e3d-a6b8-c23343d5236a" path="/var/lib/kubelet/pods/48a42e57-9048-4e3d-a6b8-c23343d5236a/volumes" Nov 04 03:01:02.051268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:02.051227 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw"] Nov 04 03:01:02.102064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:02.102028 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5f835a-9ndrw"] Nov 04 03:01:03.162363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:03.162332 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb" path="/var/lib/kubelet/pods/26cb0cf0-7aa9-4436-a2d4-dc1c4c6ff9eb/volumes" Nov 04 03:01:06.402774 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:06.402730 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:01:08.928308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.928272 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 03:01:08.929250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.929216 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" containerID="cri-o://de62a7c945b370b0a8f484cead3586bb9a09b9f87bcaa78bad12a595063ebed0" gracePeriod=30 Nov 04 03:01:08.970326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.970292 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 03:01:08.970936 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.970893 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" containerID="cri-o://03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2" gracePeriod=30 Nov 04 03:01:08.976816 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.976791 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 03:01:08.977050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:08.977024 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" containerID="cri-o://f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42" gracePeriod=30 Nov 04 03:01:09.017562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:09.017529 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 03:01:09.017743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:09.017711 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" containerID="cri-o://e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e" gracePeriod=30 Nov 04 03:01:09.548765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:09.548730 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:09.548981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:09.548811 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:14.065275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.065239 4784 generic.go:356] "Generic (PLEG): container finished" podID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerID="de62a7c945b370b0a8f484cead3586bb9a09b9f87bcaa78bad12a595063ebed0" exitCode=0 Nov 04 03:01:14.065728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.065303 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerDied","Data":"de62a7c945b370b0a8f484cead3586bb9a09b9f87bcaa78bad12a595063ebed0"} Nov 04 03:01:14.065728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.065385 4784 scope.go:122] "RemoveContainer" containerID="99174578f0e7cde51294e91604875b1a1c65e2c63d5dc0b8cf3641bc618f159a" Nov 04 03:01:14.455918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.455099 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 04 03:01:14.455918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.455176 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 04 03:01:14.541212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.540309 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 03:01:14.541212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.540415 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 03:01:14.669332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.666439 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 03:01:14.693216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.693140 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767770 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767837 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767867 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767914 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767957 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.767986 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.768039 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.768084 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.768562 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.768931 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.768990 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769078 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769116 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769181 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769325 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769375 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769457 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769484 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\") pod \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\" (UID: \"afa1207c-4bb6-47e1-ae6d-4398909b19c7\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.769535 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\") pod \"09730a8d-9021-4166-ad4b-a72626adddbe\" (UID: \"09730a8d-9021-4166-ad4b-a72626adddbe\") " Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.770419 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09730a8d-9021-4166-ad4b-a72626adddbe-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.770650 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.772269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.771017 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.778231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.778075 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.778675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.778571 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.779042 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.778883 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.779566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.779337 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:14.779851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.779700 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.780277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.779971 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.792562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.792529 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 03:01:14.795549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.795484 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.796591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.796122 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "kube-api-access-wtrzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.797435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.797394 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.798998 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.797982 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:14.803482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.803444 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.803775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.803746 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "kube-api-access-kwbdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.805857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.805824 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.813095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.811641 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token" pod "09730a8d-9021-4166-ad4b-a72626adddbe" (UID: "09730a8d-9021-4166-ad4b-a72626adddbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.813095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.811714 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo" pod "afa1207c-4bb6-47e1-ae6d-4398909b19c7" (UID: "afa1207c-4bb6-47e1-ae6d-4398909b19c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:14.834515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.834433 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 03:01:14.871783 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.871748 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.871980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.871840 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.871980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.871921 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872103 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872010 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872103 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872046 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872103 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872092 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872143 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872256 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872292 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872330 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872376 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872447 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872505 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.872734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872586 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872683 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.872734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872714 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.873058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872740 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\") pod \"6bb7646f-3fac-49f1-b138-a77250cc41d4\" (UID: \"6bb7646f-3fac-49f1-b138-a77250cc41d4\") " Nov 04 03:01:14.873058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.872802 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\") pod \"61210825-de7c-427c-87ca-ee7d63269346\" (UID: \"61210825-de7c-427c-87ca-ee7d63269346\") " Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873661 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873689 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbdz\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-kube-api-access-kwbdz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873705 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873717 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873761 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873774 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873789 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09730a8d-9021-4166-ad4b-a72626adddbe-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873803 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrzk\" (UniqueName: \"kubernetes.io/projected/afa1207c-4bb6-47e1-ae6d-4398909b19c7-kube-api-access-wtrzk\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873818 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873834 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873849 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873861 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873874 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09730a8d-9021-4166-ad4b-a72626adddbe-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873887 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09730a8d-9021-4166-ad4b-a72626adddbe-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873900 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873913 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afa1207c-4bb6-47e1-ae6d-4398909b19c7-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.873925 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afa1207c-4bb6-47e1-ae6d-4398909b19c7-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.874217 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.874491 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:14.875956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.875897 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.884534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.877339 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:14.884534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.879255 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "kube-api-access-5qln4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.884534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.879617 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.884534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.879913 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.884534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.882225 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.891091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.889842 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.891091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.890092 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.891091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.890323 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.892079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.892043 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "kube-api-access-4vgw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.892405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.892371 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:14.892823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.892799 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.893588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.893466 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:14.894139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.894105 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.895720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.895697 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token" pod "61210825-de7c-427c-87ca-ee7d63269346" (UID: "61210825-de7c-427c-87ca-ee7d63269346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.895827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.895790 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy" pod "6bb7646f-3fac-49f1-b138-a77250cc41d4" (UID: "6bb7646f-3fac-49f1-b138-a77250cc41d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976483 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976513 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976528 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7646f-3fac-49f1-b138-a77250cc41d4-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976545 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976558 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61210825-de7c-427c-87ca-ee7d63269346-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976571 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976585 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976598 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61210825-de7c-427c-87ca-ee7d63269346-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976612 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976624 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976636 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976649 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976662 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976674 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976686 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qln4\" (UniqueName: \"kubernetes.io/projected/6bb7646f-3fac-49f1-b138-a77250cc41d4-kube-api-access-5qln4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976750 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61210825-de7c-427c-87ca-ee7d63269346-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976777 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb7646f-3fac-49f1-b138-a77250cc41d4-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.984712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:14.976816 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgw4\" (UniqueName: \"kubernetes.io/projected/61210825-de7c-427c-87ca-ee7d63269346-kube-api-access-4vgw4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:15.079058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.079024 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" event={"ID":"afa1207c-4bb6-47e1-ae6d-4398909b19c7","Type":"ContainerDied","Data":"c9717b53faeee5b289bb7f93d20d96f3d6904ca9d5653b2217c8708083f1ec61"} Nov 04 03:01:15.079058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.079052 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v" Nov 04 03:01:15.079838 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.079097 4784 scope.go:122] "RemoveContainer" containerID="de62a7c945b370b0a8f484cead3586bb9a09b9f87bcaa78bad12a595063ebed0" Nov 04 03:01:15.080862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.080838 4784 generic.go:356] "Generic (PLEG): container finished" podID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerID="f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42" exitCode=0 Nov 04 03:01:15.080953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.080886 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" Nov 04 03:01:15.081056 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.081016 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" event={"ID":"6bb7646f-3fac-49f1-b138-a77250cc41d4","Type":"ContainerDied","Data":"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42"} Nov 04 03:01:15.081254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.081055 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm" event={"ID":"6bb7646f-3fac-49f1-b138-a77250cc41d4","Type":"ContainerDied","Data":"1b32058262e2d89676ad8c4236e798919eb1b2ff480ec99bdee816419cb0f434"} Nov 04 03:01:15.086774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.086630 4784 generic.go:356] "Generic (PLEG): container finished" podID="61210825-de7c-427c-87ca-ee7d63269346" containerID="e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e" exitCode=0 Nov 04 03:01:15.086774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.086752 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" Nov 04 03:01:15.089772 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.088797 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" event={"ID":"61210825-de7c-427c-87ca-ee7d63269346","Type":"ContainerDied","Data":"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e"} Nov 04 03:01:15.089772 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.088910 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw" event={"ID":"61210825-de7c-427c-87ca-ee7d63269346","Type":"ContainerDied","Data":"6659581f8768dcca97d7689428b04dad7b18b508d6c55a14ccc7cbf2a3f6f147"} Nov 04 03:01:15.105366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.105333 4784 generic.go:356] "Generic (PLEG): container finished" podID="09730a8d-9021-4166-ad4b-a72626adddbe" containerID="03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2" exitCode=0 Nov 04 03:01:15.105516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.105433 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" event={"ID":"09730a8d-9021-4166-ad4b-a72626adddbe","Type":"ContainerDied","Data":"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2"} Nov 04 03:01:15.105516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.105485 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" event={"ID":"09730a8d-9021-4166-ad4b-a72626adddbe","Type":"ContainerDied","Data":"068f6e7ec83686704e42bda27d6f87508f0cd1896ff16e59b87ed5a0f249437b"} Nov 04 03:01:15.105625 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.105593 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz" Nov 04 03:01:15.115301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.114751 4784 scope.go:122] "RemoveContainer" containerID="f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42" Nov 04 03:01:15.136599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.136568 4784 scope.go:122] "RemoveContainer" containerID="f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42" Nov 04 03:01:15.136994 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:15.136968 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42\": container with ID starting with f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42 not found: ID does not exist" containerID="f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42" Nov 04 03:01:15.137118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.137006 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42"} err="failed to get container status \"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42\": rpc error: code = NotFound desc = could not find container \"f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42\": container with ID starting with f10fff8d5b3a9d3abf5ecdc1764e5b2f4d906e937c9a86350e67159e569eaf42 not found: ID does not exist" Nov 04 03:01:15.137118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.137033 4784 scope.go:122] "RemoveContainer" containerID="e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e" Nov 04 03:01:15.153269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.153180 4784 scope.go:122] "RemoveContainer" containerID="e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e" Nov 04 03:01:15.153817 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:15.153754 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e\": container with ID starting with e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e not found: ID does not exist" containerID="e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e" Nov 04 03:01:15.153944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.153832 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e"} err="failed to get container status \"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e\": rpc error: code = NotFound desc = could not find container \"e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e\": container with ID starting with e91ae42601f546373a0db8c5907e6447aad05935db7b2cf299731b99a00fd10e not found: ID does not exist" Nov 04 03:01:15.153944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.153859 4784 scope.go:122] "RemoveContainer" containerID="03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2" Nov 04 03:01:15.169872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.169511 4784 scope.go:122] "RemoveContainer" containerID="03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2" Nov 04 03:01:15.169872 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:15.169856 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2\": container with ID starting with 03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2 not found: ID does not exist" containerID="03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2" Nov 04 03:01:15.170001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.169893 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2"} err="failed to get container status \"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2\": rpc error: code = NotFound desc = could not find container \"03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2\": container with ID starting with 03be25ff43f823f61865e2277c0ced1cd7442a2caf5673b2b2196caa872bc6f2 not found: ID does not exist" Nov 04 03:01:15.435675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.435642 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 03:01:15.442558 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.442526 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-frgsz"] Nov 04 03:01:15.554430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.554381 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 03:01:15.560082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.560054 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-4qkqm"] Nov 04 03:01:15.673279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.673250 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 03:01:15.677772 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.677731 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-x449v"] Nov 04 03:01:15.791226 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.791130 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 03:01:15.795880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:15.795854 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-db5nw"] Nov 04 03:01:17.163784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:17.163745 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" path="/var/lib/kubelet/pods/09730a8d-9021-4166-ad4b-a72626adddbe/volumes" Nov 04 03:01:17.164331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:17.164314 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61210825-de7c-427c-87ca-ee7d63269346" path="/var/lib/kubelet/pods/61210825-de7c-427c-87ca-ee7d63269346/volumes" Nov 04 03:01:17.164841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:17.164822 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" path="/var/lib/kubelet/pods/6bb7646f-3fac-49f1-b138-a77250cc41d4/volumes" Nov 04 03:01:17.165778 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:17.165682 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" path="/var/lib/kubelet/pods/afa1207c-4bb6-47e1-ae6d-4398909b19c7/volumes" Nov 04 03:01:19.695296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.695239 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:01:19.697029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697003 4784 cpu_manager.go:414] "containerMap: removing container" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" Nov 04 03:01:19.697252 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697172 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" Nov 04 03:01:19.697395 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697382 4784 cpu_manager.go:414] "containerMap: removing container" podUID="dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" containerName="revoke-keys" Nov 04 03:01:19.697472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697462 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" containerName="revoke-keys" Nov 04 03:01:19.697609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697600 4784 cpu_manager.go:414] "containerMap: removing container" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:01:19.697689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697680 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ce3d8f-ccb9-4260-9e43-c6fee7fdb698" containerName="cleanup" Nov 04 03:01:19.697779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.697770 4784 cpu_manager.go:414] "containerMap: removing container" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" Nov 04 03:01:19.706265 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706236 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" Nov 04 03:01:19.706461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706450 4784 cpu_manager.go:414] "containerMap: removing container" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.706538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706529 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.706614 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706605 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" Nov 04 03:01:19.706689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706680 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" Nov 04 03:01:19.706761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706753 4784 cpu_manager.go:414] "containerMap: removing container" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" Nov 04 03:01:19.706829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706820 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" Nov 04 03:01:19.706912 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706904 4784 cpu_manager.go:414] "containerMap: removing container" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.706984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.706975 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.707063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707055 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" Nov 04 03:01:19.707135 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707125 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" Nov 04 03:01:19.707481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707466 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a89c764-c3b5-4086-898f-f0a02b8e23ef" containerName="cleanup" Nov 04 03:01:19.707591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707581 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1f82d3-a565-4cad-831c-fb08f3c288ec" containerName="cleanup" Nov 04 03:01:19.707689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707678 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5eb4f14-4660-4923-b71b-318b8e1fdbaf" containerName="cleanup" Nov 04 03:01:19.707781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707773 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c133df67-edc1-430e-b578-d3ec68bf5fed" containerName="cleanup" Nov 04 03:01:19.707864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707856 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cb22f0-97b4-42af-8f2e-6adc4b331bdb" containerName="istio-proxy" Nov 04 03:01:19.707933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707922 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.708001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.707993 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4f7bc-152f-477b-827f-2099474380bc" containerName="istio-proxy" Nov 04 03:01:19.708062 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.708056 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="61210825-de7c-427c-87ca-ee7d63269346" containerName="istio-proxy" Nov 04 03:01:19.708132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.708124 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa1207c-4bb6-47e1-ae6d-4398909b19c7" containerName="istio-proxy" Nov 04 03:01:19.708206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.708182 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb7646f-3fac-49f1-b138-a77250cc41d4" containerName="istio-proxy" Nov 04 03:01:19.708290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.708282 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="09730a8d-9021-4166-ad4b-a72626adddbe" containerName="istio-proxy" Nov 04 03:01:19.708355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.708347 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" containerName="revoke-keys" Nov 04 03:01:19.718912 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.718881 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810510 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810566 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810680 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810729 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810818 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b422\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.810848 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.811041 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.811072 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.811472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.811115 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.831461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.831260 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:01:19.846823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.846791 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-551e2b-openshift-default-dockercfg-cbb4r\"" Nov 04 03:01:19.912665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912617 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.912901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912676 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.912901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912730 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.912901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912823 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.912901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912863 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.913154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.912983 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.913154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.913036 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.913154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.913131 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b422\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.913355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.913165 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.921233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.913716 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.928463 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.929044 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.929521 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.929780 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.929998 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:19.932269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:19.931541 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:20.037372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.032831 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:20.037372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.033779 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b422\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\") pod \"e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:20.221966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.221919 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:20.833275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.833214 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:01:20.836803 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.836755 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:20.837026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.837005 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:20.837166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:20.837146 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:21.287502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:21.287461 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerStarted","Data":"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd"} Nov 04 03:01:21.287879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:21.287852 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerStarted","Data":"b0c10234a1bd4cb88fa9197cfe0de9790c0b20c0ac8dd97dc61fda88169ddd9a"} Nov 04 03:01:21.491218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:21.490736 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podStartSLOduration=2.490716702 podStartE2EDuration="2.490716702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:21.478869018 +0000 UTC m=+1728.445370010" watchObservedRunningTime="2026-11-04 03:01:21.490716702 +0000 UTC m=+1728.457217694" Nov 04 03:01:22.227368 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:22.227324 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:22.229174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:22.229144 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:01:22.229493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:22.229440 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:01:23.230486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:23.230448 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:23.317057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:23.316756 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:23.318251 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:23.318130 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:01:24.370289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.370256 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng"] Nov 04 03:01:24.422961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.422930 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5fde04ee-b75ng"] Nov 04 03:01:24.435044 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.435014 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 03:01:24.435322 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.435266 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" containerID="cri-o://bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2" gracePeriod=30 Nov 04 03:01:24.897741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.897362 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:01:24.897741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:24.897440 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:25.200031 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:25.199993 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe384e7-8c53-4fa6-9f91-1dc3985ad2de" path="/var/lib/kubelet/pods/dbe384e7-8c53-4fa6-9f91-1dc3985ad2de/volumes" Nov 04 03:01:28.393909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.393809 4784 generic.go:356] "Generic (PLEG): container finished" podID="efb0907e-7ded-4261-971b-6149ac28fb07" containerID="250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4" exitCode=0 Nov 04 03:01:28.393909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.393858 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerDied","Data":"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4"} Nov 04 03:01:28.394573 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.394493 4784 scope.go:122] "RemoveContainer" containerID="250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4" Nov 04 03:01:28.396802 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.396740 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:28.397001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.396874 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:28.397001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:28.396929 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:01:29.405538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.405506 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerStarted","Data":"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81"} Nov 04 03:01:29.689207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.688306 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:01:29.689207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.688479 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:29.689207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.688525 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:29.815523 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.815492 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 03:01:29.925707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.925676 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.925892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.925843 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.925949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.925911 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.925963 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:29.926058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926018 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926117 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926063 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926177 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926124 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926177 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926155 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926392 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926372 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.926564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926487 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:29.926564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926423 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:29.926709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926687 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:29.926935 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.926807 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\") pod \"a1e24565-e859-4c16-a564-051c809fff3d\" (UID: \"a1e24565-e859-4c16-a564-051c809fff3d\") " Nov 04 03:01:29.927093 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.927038 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:29.927968 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.927941 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:29.928082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.927971 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:29.928082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.927989 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:29.928082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928003 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e24565-e859-4c16-a564-051c809fff3d-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:29.928082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928016 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:29.928548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928525 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:29.928754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928729 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:29.928914 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928896 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:29.929009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:29.928959 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r" pod "a1e24565-e859-4c16-a564-051c809fff3d" (UID: "a1e24565-e859-4c16-a564-051c809fff3d"). InnerVolumeSpecName "kube-api-access-k9d5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:30.029579 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.029545 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e24565-e859-4c16-a564-051c809fff3d-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:30.029767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.029592 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-kube-api-access-k9d5r\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:30.029767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.029603 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e24565-e859-4c16-a564-051c809fff3d-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:30.029767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.029611 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e24565-e859-4c16-a564-051c809fff3d-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:30.417018 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.416989 4784 generic.go:356] "Generic (PLEG): container finished" podID="a1e24565-e859-4c16-a564-051c809fff3d" containerID="bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2" exitCode=0 Nov 04 03:01:30.417586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.417041 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" Nov 04 03:01:30.417586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.417076 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerDied","Data":"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2"} Nov 04 03:01:30.417586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.417148 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z" event={"ID":"a1e24565-e859-4c16-a564-051c809fff3d","Type":"ContainerDied","Data":"43a6ad03cbecd78bfc377413e9628f46d1c7d709bf27a826fdfd1c149c3870b5"} Nov 04 03:01:30.417586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.417183 4784 scope.go:122] "RemoveContainer" containerID="bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2" Nov 04 03:01:30.430849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.430830 4784 scope.go:122] "RemoveContainer" containerID="02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c" Nov 04 03:01:30.442932 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.442911 4784 scope.go:122] "RemoveContainer" containerID="bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2" Nov 04 03:01:30.443305 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:30.443283 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2\": container with ID starting with bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2 not found: ID does not exist" containerID="bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2" Nov 04 03:01:30.443392 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.443324 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2"} err="failed to get container status \"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2\": rpc error: code = NotFound desc = could not find container \"bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2\": container with ID starting with bcf5dfd92ddd4bc6f7f84bad70b78179e2a96aa3553a6d0aaeb4597e26b5a8f2 not found: ID does not exist" Nov 04 03:01:30.443392 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.443345 4784 scope.go:122] "RemoveContainer" containerID="02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c" Nov 04 03:01:30.443633 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:30.443614 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c\": container with ID starting with 02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c not found: ID does not exist" containerID="02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c" Nov 04 03:01:30.443686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.443639 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c"} err="failed to get container status \"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c\": rpc error: code = NotFound desc = could not find container \"02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c\": container with ID starting with 02f91d6317f9a2f0f591d6ca2e818660428aba3a561bfb6e7efdc088b6c1e93c not found: ID does not exist" Nov 04 03:01:30.590656 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.590621 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 03:01:30.596361 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.596327 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5fde04ee-openshift-default-7d7db879c-bxx4z"] Nov 04 03:01:30.680700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.680610 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:30.680700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:30.680686 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:31.187541 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:31.187506 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e24565-e859-4c16-a564-051c809fff3d" path="/var/lib/kubelet/pods/a1e24565-e859-4c16-a564-051c809fff3d/volumes" Nov 04 03:01:31.682634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:31.682590 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:31.683250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:31.682661 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:32.680624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:32.680587 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:32.680843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:32.680662 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:33.682031 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:33.681992 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:33.682675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:33.682640 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.411283 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412430 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412452 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412488 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412497 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412745 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.414139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.412762 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e24565-e859-4c16-a564-051c809fff3d" containerName="istio-proxy" Nov 04 03:01:34.418736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.415571 4784 scope.go:122] "RemoveContainer" containerID="cb044b86d65039614fee380599616823c75bf413b11dbc3d18706a8ff323d4d1" Nov 04 03:01:34.420072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.420051 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.480067 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.480036 4784 scope.go:122] "RemoveContainer" containerID="6e32c5cb21bf213ff518cfabaca836a9d6a6fe30420d42d911964ffb784b4103" Nov 04 03:01:34.495267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.489763 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:01:34.514584 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.514521 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-551e2b\"" Nov 04 03:01:34.619165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.619038 4784 scope.go:122] "RemoveContainer" containerID="332d78da0c78348eda88eeee9682ba1db408f3209d2644e521131ebf29ce5dff" Nov 04 03:01:34.619165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.619160 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zsnv\" (UniqueName: \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.619447 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.619381 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.704270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.684028 4784 patch_prober.go:29] interesting pod/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 03:01:34.704270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.684088 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 03:01:34.758551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.758505 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.762027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.761994 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zsnv\" (UniqueName: \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.763862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.763818 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.764814 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.764796 4784 scope.go:122] "RemoveContainer" containerID="b929e2de57d604806013a657c8ff87169ee45cf772e6bbaade496783e84be323" Nov 04 03:01:34.879240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.864564 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x"] Nov 04 03:01:34.921349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.921313 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zsnv\" (UniqueName: \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\") pod \"payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:34.946083 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.946049 4784 scope.go:122] "RemoveContainer" containerID="584b026aa1a4bbae50786b5ea1f7260d57fc854f6cba276b7e415a51e2e0bee0" Nov 04 03:01:34.953832 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.953802 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x"] Nov 04 03:01:34.954151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.954135 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:34.977756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.977733 4784 scope.go:122] "RemoveContainer" containerID="fe84aa48ae3afd50cd4627a4099238ed37f0a35e5dbbbb2b0b7a6e100b798634" Nov 04 03:01:34.989349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.989308 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:34.990111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.990082 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:34.990316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:34.990296 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz44d\" (UniqueName: \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.009623 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.009593 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:01:35.010133 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.010111 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:01:35.010437 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.010421 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:01:35.010684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.010668 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:01:35.017419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.017402 4784 scope.go:122] "RemoveContainer" containerID="61c55de286401eaa20d3c2d211e5a80820d3a8617de2b236355de00ba66b55b5" Nov 04 03:01:35.033639 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.033561 4784 scope.go:122] "RemoveContainer" containerID="dbed71d1b30ab12907a531f2d4c71e4a661438c52c0c3eb7d023061e4c035ee5" Nov 04 03:01:35.048702 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.048669 4784 scope.go:122] "RemoveContainer" containerID="ed53aa66bdcef30ce7ac9627dc1b05a5f3d172ce3dc10af547c267a8d7f9df37" Nov 04 03:01:35.060015 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.059990 4784 scope.go:122] "RemoveContainer" containerID="680f6b167f705c8a3093a4f133625b5af76bdb20eee3d33a553cbcd6294b7b38" Nov 04 03:01:35.075397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.075369 4784 scope.go:122] "RemoveContainer" containerID="feecfe2d2f21cb8e930547d82a4731fcb2ca78ff8b9ff17083a0b6d5b4f91c84" Nov 04 03:01:35.089817 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.089789 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:35.091455 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.091434 4784 scope.go:122] "RemoveContainer" containerID="9a69d3265de8f688b0142cb134fe040bdcd3d200bc10cf7e3abdafddcda4c25e" Nov 04 03:01:35.092183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.092156 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.092524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.092472 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.092524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.092500 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz44d\" (UniqueName: \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.093241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.093219 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.095543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.095505 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.141774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.141620 4784 scope.go:122] "RemoveContainer" containerID="1828d0d1dd330db3499b4705fda7d7983b188d7b15705e29c31e680b26a1be3b" Nov 04 03:01:35.162116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.161960 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz44d\" (UniqueName: \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\") pod \"maas-api-revoke-sub-6d316993-6857x\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.214800 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.213673 4784 scope.go:122] "RemoveContainer" containerID="657f40b58e60e146f8f4cded3aa9517eaedc0dc39934ece6ea241fb089796e01" Nov 04 03:01:35.233140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.233061 4784 scope.go:122] "RemoveContainer" containerID="569aaab9898221af44895da41b7bbd8f3887cbcbadd9e88baf780df1687aa04a" Nov 04 03:01:35.252671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.252643 4784 scope.go:122] "RemoveContainer" containerID="9b96ad73177f7fcc61e99df3e01bd666d062dce8f5c2839b4fded5923fc47e31" Nov 04 03:01:35.292636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.292614 4784 scope.go:122] "RemoveContainer" containerID="b6cff63b70a9c0cb8feb5c429688f95560c3c0f95e0f45dc561d02d1d99a946e" Nov 04 03:01:35.305649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.305622 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:35.501975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.501944 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:01:35.502456 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:01:35.502429 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db5e7bc_970b_4751_b375_884f867fe031.slice/crio-9d465f0ee46d5456dcad20004a4e766b6b3edbbd1ad2fb50c43b149e3f51779a WatchSource:0}: Error finding container 9d465f0ee46d5456dcad20004a4e766b6b3edbbd1ad2fb50c43b149e3f51779a: Status 404 returned error can't find the container with id 9d465f0ee46d5456dcad20004a4e766b6b3edbbd1ad2fb50c43b149e3f51779a Nov 04 03:01:35.633394 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.633340 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" event={"ID":"0db5e7bc-970b-4751-b375-884f867fe031","Type":"ContainerStarted","Data":"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6"} Nov 04 03:01:35.633586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.633413 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" event={"ID":"0db5e7bc-970b-4751-b375-884f867fe031","Type":"ContainerStarted","Data":"9d465f0ee46d5456dcad20004a4e766b6b3edbbd1ad2fb50c43b149e3f51779a"} Nov 04 03:01:35.633586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.633447 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:35.685461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.685432 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:01:35.685752 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.685731 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:01:35.686541 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.686515 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:01:35.726537 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:01:35.726497 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa54205_e485_4ad4_9cea_75a0b221aa19.slice/crio-3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc WatchSource:0}: Error finding container 3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc: Status 404 returned error can't find the container with id 3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc Nov 04 03:01:35.727811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.727786 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x"] Nov 04 03:01:35.754247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:35.754173 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" podStartSLOduration=1.7541548580000002 podStartE2EDuration="1.754154858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:35.750887042 +0000 UTC m=+1742.717388036" watchObservedRunningTime="2026-11-04 03:01:35.754154858 +0000 UTC m=+1742.720655853" Nov 04 03:01:36.402340 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:36.402251 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:01:36.644548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:36.644508 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" event={"ID":"3fa54205-e485-4ad4-9cea-75a0b221aa19","Type":"ContainerStarted","Data":"6cfb367f752c3ba16d9c0793209b50dec560f90b28087e034f277ee64a1d9904"} Nov 04 03:01:36.644744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:36.644588 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" event={"ID":"3fa54205-e485-4ad4-9cea-75a0b221aa19","Type":"ContainerStarted","Data":"3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc"} Nov 04 03:01:36.765570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:36.765517 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" podStartSLOduration=2.765501276 podStartE2EDuration="2.765501276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:36.762838102 +0000 UTC m=+1743.729339094" watchObservedRunningTime="2026-11-04 03:01:36.765501276 +0000 UTC m=+1743.732002265" Nov 04 03:01:37.654661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:37.654621 4784 generic.go:356] "Generic (PLEG): container finished" podID="3fa54205-e485-4ad4-9cea-75a0b221aa19" containerID="6cfb367f752c3ba16d9c0793209b50dec560f90b28087e034f277ee64a1d9904" exitCode=0 Nov 04 03:01:37.654850 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:37.654700 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" event={"ID":"3fa54205-e485-4ad4-9cea-75a0b221aa19","Type":"ContainerDied","Data":"6cfb367f752c3ba16d9c0793209b50dec560f90b28087e034f277ee64a1d9904"} Nov 04 03:01:38.851382 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.851355 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:38.978076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.977991 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\") pod \"3fa54205-e485-4ad4-9cea-75a0b221aa19\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " Nov 04 03:01:38.978076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.978032 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\") pod \"3fa54205-e485-4ad4-9cea-75a0b221aa19\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " Nov 04 03:01:38.978459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.978321 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\" (UniqueName: \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\") pod \"3fa54205-e485-4ad4-9cea-75a0b221aa19\" (UID: \"3fa54205-e485-4ad4-9cea-75a0b221aa19\") " Nov 04 03:01:38.978550 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.978522 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca" pod "3fa54205-e485-4ad4-9cea-75a0b221aa19" (UID: "3fa54205-e485-4ad4-9cea-75a0b221aa19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:38.979045 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.979024 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:38.980578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.980554 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca" pod "3fa54205-e485-4ad4-9cea-75a0b221aa19" (UID: "3fa54205-e485-4ad4-9cea-75a0b221aa19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:01:38.980877 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:38.980858 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d" pod "3fa54205-e485-4ad4-9cea-75a0b221aa19" (UID: "3fa54205-e485-4ad4-9cea-75a0b221aa19"). InnerVolumeSpecName "kube-api-access-mz44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:39.034236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.034181 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 03:01:39.034487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.034459 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" containerID="cri-o://3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf" gracePeriod=30 Nov 04 03:01:39.083283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.083248 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz44d\" (UniqueName: \"kubernetes.io/projected/3fa54205-e485-4ad4-9cea-75a0b221aa19-kube-api-access-mz44d\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:39.083283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.083281 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa54205-e485-4ad4-9cea-75a0b221aa19-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:39.674483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.674456 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" Nov 04 03:01:39.674644 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.674457 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x" event={"ID":"3fa54205-e485-4ad4-9cea-75a0b221aa19","Type":"ContainerDied","Data":"3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc"} Nov 04 03:01:39.674644 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:39.674582 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3412e86be2e5859322ecc4f486667aa054be1da37a0c7ffd27846050079ef0dc" Nov 04 03:01:41.987966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:41.987212 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:01:41.988470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:41.988386 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3fa54205-e485-4ad4-9cea-75a0b221aa19" containerName="revoke-keys" Nov 04 03:01:41.988470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:41.988404 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa54205-e485-4ad4-9cea-75a0b221aa19" containerName="revoke-keys" Nov 04 03:01:41.988597 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:41.988518 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa54205-e485-4ad4-9cea-75a0b221aa19" containerName="revoke-keys" Nov 04 03:01:41.991397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:41.991378 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.039247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.036463 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:01:42.136213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.134665 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-4b2bf4\"" Nov 04 03:01:42.136213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.134821 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-4b2bf4-dockercfg-5vfkx\"" Nov 04 03:01:42.144347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.142838 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdc4\" (UniqueName: \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.144347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.142956 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.147351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.147323 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:01:42.175222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.174032 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.210498 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.210402 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:01:42.268751 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.268655 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.269978 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.269943 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.270159 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.270141 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wx5\" (UniqueName: \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.270357 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.270342 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdc4\" (UniqueName: \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.270752 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.269788 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.333294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.333258 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdc4\" (UniqueName: \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\") pod \"payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.371984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.371948 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wx5\" (UniqueName: \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.372246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.372171 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.372875 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.372848 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.440346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.440313 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wx5\" (UniqueName: \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\") pod \"payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.503915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.503870 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:42.609457 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.609424 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:42.886941 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:42.886910 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:01:42.887082 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:01:42.887033 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef06c0b7_2af6_4949_9777_0c71c78e65c2.slice/crio-ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112 WatchSource:0}: Error finding container ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112: Status 404 returned error can't find the container with id ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112 Nov 04 03:01:43.012761 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:01:43.012728 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc78c6d_584a_4bb6_aceb_e986fbae8e2e.slice/crio-a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba WatchSource:0}: Error finding container a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba: Status 404 returned error can't find the container with id a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba Nov 04 03:01:43.013834 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.013806 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:01:43.745076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.745029 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" event={"ID":"ef06c0b7-2af6-4949-9777-0c71c78e65c2","Type":"ContainerStarted","Data":"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e"} Nov 04 03:01:43.745291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.745096 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" event={"ID":"ef06c0b7-2af6-4949-9777-0c71c78e65c2","Type":"ContainerStarted","Data":"ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112"} Nov 04 03:01:43.745291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.745181 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:43.747050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.747022 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" event={"ID":"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e","Type":"ContainerStarted","Data":"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd"} Nov 04 03:01:43.747172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.747068 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:43.747172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.747091 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" event={"ID":"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e","Type":"ContainerStarted","Data":"a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba"} Nov 04 03:01:43.865025 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.864972 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" podStartSLOduration=1.864957139 podStartE2EDuration="1.864957139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:43.864888494 +0000 UTC m=+1750.831389485" watchObservedRunningTime="2026-11-04 03:01:43.864957139 +0000 UTC m=+1750.831458127" Nov 04 03:01:43.998235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:43.996804 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" podStartSLOduration=2.996779998 podStartE2EDuration="2.996779998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:43.989680167 +0000 UTC m=+1750.956181194" watchObservedRunningTime="2026-11-04 03:01:43.996779998 +0000 UTC m=+1750.963280991" Nov 04 03:01:44.401582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.401554 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 03:01:44.512472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512439 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512570 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512633 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512654 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512703 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512710 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:44.512879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512727 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512761 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.512879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512845 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.513168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512950 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:44.513168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512967 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:44.513168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512979 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:44.513168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.512960 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\") pod \"08f62f36-e794-4802-aa54-d2a93106fd9e\" (UID: \"08f62f36-e794-4802-aa54-d2a93106fd9e\") " Nov 04 03:01:44.513168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.513154 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:44.514256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.514092 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.514256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.514127 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62f36-e794-4802-aa54-d2a93106fd9e-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.514256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.514144 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.514256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.514159 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.514256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.514173 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.515368 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.515346 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:44.515471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.515425 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:44.515471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.515456 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "kube-api-access-zh5v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:44.515729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.515715 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo" pod "08f62f36-e794-4802-aa54-d2a93106fd9e" (UID: "08f62f36-e794-4802-aa54-d2a93106fd9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:44.616346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.616308 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.616346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.616350 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5v4\" (UniqueName: \"kubernetes.io/projected/08f62f36-e794-4802-aa54-d2a93106fd9e-kube-api-access-zh5v4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.616552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.616367 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.616552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.616382 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62f36-e794-4802-aa54-d2a93106fd9e-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:44.757425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.757384 4784 generic.go:356] "Generic (PLEG): container finished" podID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerID="3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf" exitCode=0 Nov 04 03:01:44.757601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.757435 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" Nov 04 03:01:44.757601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.757460 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" event={"ID":"08f62f36-e794-4802-aa54-d2a93106fd9e","Type":"ContainerDied","Data":"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf"} Nov 04 03:01:44.757601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.757547 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw" event={"ID":"08f62f36-e794-4802-aa54-d2a93106fd9e","Type":"ContainerDied","Data":"9ae60dbf42e7dae0aa8b0979104c29d78d4a0168649a6b1c97bf282d1c1d12d3"} Nov 04 03:01:44.757601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.757572 4784 scope.go:122] "RemoveContainer" containerID="3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf" Nov 04 03:01:44.773044 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.773024 4784 scope.go:122] "RemoveContainer" containerID="3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf" Nov 04 03:01:44.773378 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:01:44.773356 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf\": container with ID starting with 3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf not found: ID does not exist" containerID="3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf" Nov 04 03:01:44.773432 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.773387 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf"} err="failed to get container status \"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf\": rpc error: code = NotFound desc = could not find container \"3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf\": container with ID starting with 3e6c7997170925a8bf59acf263a41a7093c0d4cd32e5da5fb9e18ac972819ccf not found: ID does not exist" Nov 04 03:01:44.991285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:44.991248 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 03:01:45.002384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:45.002231 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-f9flw"] Nov 04 03:01:45.185384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:45.185350 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" path="/var/lib/kubelet/pods/08f62f36-e794-4802-aa54-d2a93106fd9e/volumes" Nov 04 03:01:46.648767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:46.648742 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:01:47.484217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:47.476738 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:01:47.484217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:47.476814 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:52.931152 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:52.931110 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 03:01:52.931794 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:52.931362 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" containerID="cri-o://dc74bb1907848c89871e134547ff175d53212ba7bab4f8812d0fb566dda0444e" gracePeriod=30 Nov 04 03:01:54.761126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:54.761093 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:01:54.761621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:54.761169 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:01:54.984969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:54.984932 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:54.985175 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:54.985002 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:56.062397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:56.062357 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 03:01:58.179439 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.179058 4784 generic.go:356] "Generic (PLEG): container finished" podID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerID="dc74bb1907848c89871e134547ff175d53212ba7bab4f8812d0fb566dda0444e" exitCode=0 Nov 04 03:01:58.179439 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.179362 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" event={"ID":"29de197f-6a69-4a00-91c5-fc25172fc4f4","Type":"ContainerDied","Data":"dc74bb1907848c89871e134547ff175d53212ba7bab4f8812d0fb566dda0444e"} Nov 04 03:01:58.227900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.227873 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 03:01:58.344654 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344626 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.344848 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344728 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.344848 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344777 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.344974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344861 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.344974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344929 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.344974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344942 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.345120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.344977 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.345174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345116 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.345174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345152 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.345301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345165 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.345301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345258 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.345403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345344 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\") pod \"29de197f-6a69-4a00-91c5-fc25172fc4f4\" (UID: \"29de197f-6a69-4a00-91c5-fc25172fc4f4\") " Nov 04 03:01:58.345685 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345653 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:58.345790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.345764 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.346462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.346435 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.346564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.346467 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.346564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.346484 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29de197f-6a69-4a00-91c5-fc25172fc4f4-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.346564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.346502 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.346564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.346517 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.347164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.347145 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.347803 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.347779 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "kube-api-access-njlbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.347909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.347836 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:58.347909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.347858 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token" pod "29de197f-6a69-4a00-91c5-fc25172fc4f4" (UID: "29de197f-6a69-4a00-91c5-fc25172fc4f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.448339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.448300 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.448339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.448334 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.448339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.448345 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-kube-api-access-njlbv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.448669 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:58.448358 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29de197f-6a69-4a00-91c5-fc25172fc4f4-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:59.189205 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:59.189162 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" Nov 04 03:01:59.189663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:59.189213 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h" event={"ID":"29de197f-6a69-4a00-91c5-fc25172fc4f4","Type":"ContainerDied","Data":"0558f2dc11d68b2c4d66252f1a1feac948ad25d0b4221e167ba9fc86d6bee2ef"} Nov 04 03:01:59.189663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:59.189293 4784 scope.go:122] "RemoveContainer" containerID="dc74bb1907848c89871e134547ff175d53212ba7bab4f8812d0fb566dda0444e" Nov 04 03:01:59.419823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:59.419783 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 03:01:59.465784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:01:59.465704 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-ck68h"] Nov 04 03:02:01.063648 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.063561 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" podUID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" containerName="payload-processing" containerID="cri-o://38b2ab2c67b77351e81b7cc34643a38a92785e458c2ae3872072dfb653917046" gracePeriod=25 Nov 04 03:02:01.162939 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.162906 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" path="/var/lib/kubelet/pods/29de197f-6a69-4a00-91c5-fc25172fc4f4/volumes" Nov 04 03:02:01.209878 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.209842 4784 generic.go:356] "Generic (PLEG): container finished" podID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" containerID="38b2ab2c67b77351e81b7cc34643a38a92785e458c2ae3872072dfb653917046" exitCode=0 Nov 04 03:02:01.210086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.209921 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" event={"ID":"f6aec14e-54b3-44c2-8555-9dad9a6845ce","Type":"ContainerDied","Data":"38b2ab2c67b77351e81b7cc34643a38a92785e458c2ae3872072dfb653917046"} Nov 04 03:02:01.359345 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.359317 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 03:02:01.495316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.495274 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\") pod \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " Nov 04 03:02:01.495534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.495365 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\" (UniqueName: \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\") pod \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\" (UID: \"f6aec14e-54b3-44c2-8555-9dad9a6845ce\") " Nov 04 03:02:01.495813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.495781 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume" pod "f6aec14e-54b3-44c2-8555-9dad9a6845ce" (UID: "f6aec14e-54b3-44c2-8555-9dad9a6845ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:02:01.496172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.496149 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6aec14e-54b3-44c2-8555-9dad9a6845ce-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:01.498440 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.498411 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z" pod "f6aec14e-54b3-44c2-8555-9dad9a6845ce" (UID: "f6aec14e-54b3-44c2-8555-9dad9a6845ce"). InnerVolumeSpecName "kube-api-access-wld8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:02:01.597798 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:01.597705 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wld8z\" (UniqueName: \"kubernetes.io/projected/f6aec14e-54b3-44c2-8555-9dad9a6845ce-kube-api-access-wld8z\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:02.220417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:02.220384 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" Nov 04 03:02:02.220880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:02.220413 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t" event={"ID":"f6aec14e-54b3-44c2-8555-9dad9a6845ce","Type":"ContainerDied","Data":"562936bb8cb1f25965ccf650a0b2aa754f7c311cecaa3e92f1cc2ba42059d43d"} Nov 04 03:02:02.220880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:02.220481 4784 scope.go:122] "RemoveContainer" containerID="38b2ab2c67b77351e81b7cc34643a38a92785e458c2ae3872072dfb653917046" Nov 04 03:02:02.390705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:02.390666 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 03:02:02.441768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:02.441734 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4d0787-5558ff6d68-hdw6t"] Nov 04 03:02:03.163023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:03.162987 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" path="/var/lib/kubelet/pods/f6aec14e-54b3-44c2-8555-9dad9a6845ce/volumes" Nov 04 03:02:06.402630 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:02:06.402590 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:02:19.872076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.872019 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:02:19.873290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873264 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" containerName="payload-processing" Nov 04 03:02:19.873410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873292 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" containerName="payload-processing" Nov 04 03:02:19.873410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873324 4784 cpu_manager.go:414] "containerMap: removing container" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" Nov 04 03:02:19.873410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873331 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" Nov 04 03:02:19.873410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873354 4784 cpu_manager.go:414] "containerMap: removing container" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" Nov 04 03:02:19.873410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873362 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" Nov 04 03:02:19.873686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873577 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f62f36-e794-4802-aa54-d2a93106fd9e" containerName="istio-proxy" Nov 04 03:02:19.873686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873605 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6aec14e-54b3-44c2-8555-9dad9a6845ce" containerName="payload-processing" Nov 04 03:02:19.873686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.873618 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="29de197f-6a69-4a00-91c5-fc25172fc4f4" containerName="istio-proxy" Nov 04 03:02:19.879744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.879712 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:19.937709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:19.937663 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-569fb7-openshift-default-dockercfg-g4r9l\"" Nov 04 03:02:20.005295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.005234 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.038355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.038318 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.038747 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.038724 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.038969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.038933 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.039141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.039114 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79l7\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.048421 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.043281 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.135859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.135762 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.142171 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.142118 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.142574 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.142541 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.163669 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.163632 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:02:20.244275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.244230 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.244615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.244594 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.244806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.244788 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.244937 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.244922 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79l7\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.245083 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.245066 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.245247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.245230 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.245463 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.245444 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.245626 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.245610 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.245941 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.245918 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.248642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.248617 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.251498 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.249615 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.251690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.250134 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.251790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.250878 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.252480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.252453 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.283207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.283149 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.283690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.283661 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.313036 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.312974 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.317458 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.317426 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79l7\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\") pod \"e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:20.514318 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:20.512727 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:21.028017 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.027874 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:02:21.048709 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:02:21.048673 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcdf837_d0c4_4a46_a9eb_caf3ea268789.slice/crio-2b74170f880a802a3467f8402b4361d040098d5d3fd8063096bc894bd6e0bd40 WatchSource:0}: Error finding container 2b74170f880a802a3467f8402b4361d040098d5d3fd8063096bc894bd6e0bd40: Status 404 returned error can't find the container with id 2b74170f880a802a3467f8402b4361d040098d5d3fd8063096bc894bd6e0bd40 Nov 04 03:02:21.051648 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.051613 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:21.051756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.051697 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:21.051756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.051726 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:21.532226 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.532008 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerStarted","Data":"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f"} Nov 04 03:02:21.532226 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:21.532080 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerStarted","Data":"2b74170f880a802a3467f8402b4361d040098d5d3fd8063096bc894bd6e0bd40"} Nov 04 03:02:22.517214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:22.516963 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:22.536861 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:22.536823 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:22.548154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:22.548121 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:22.549256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:22.549238 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:02:22.678249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:22.678151 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podStartSLOduration=3.678128722 podStartE2EDuration="3.678128722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:21.723239797 +0000 UTC m=+1788.689740790" watchObservedRunningTime="2026-11-04 03:02:22.678128722 +0000 UTC m=+1789.644629715" Nov 04 03:02:30.923363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:30.923324 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:02:30.928355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:30.928333 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:30.984952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:30.984917 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:02:31.001383 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.001345 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsqc\" (UniqueName: \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.001563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.001526 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.031260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.031229 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-569fb7-dockercfg-wllnp\"" Nov 04 03:02:31.031487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.031338 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-569fb7\"" Nov 04 03:02:31.208682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.181852 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsqc\" (UniqueName: \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.208682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.198525 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.226669 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.220725 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.323917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.323867 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsqc\" (UniqueName: \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\") pod \"payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.326712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.326663 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:02:31.332813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.332781 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:02:31.332953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.332923 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.449851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.449806 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.450541 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.450217 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.545451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.545367 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:31.552482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.552446 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.552615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.552527 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.553248 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.553225 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.615367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.615330 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\") pod \"payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.657771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.657528 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:31.933924 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:02:31.933889 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda786813b_3dfa_405d_be13_45061bf3e640.slice/crio-412b8e5097a26ca0a45b754b278595d8fb1dc74fde09e70c844155afd3bc1fc8 WatchSource:0}: Error finding container 412b8e5097a26ca0a45b754b278595d8fb1dc74fde09e70c844155afd3bc1fc8: Status 404 returned error can't find the container with id 412b8e5097a26ca0a45b754b278595d8fb1dc74fde09e70c844155afd3bc1fc8 Nov 04 03:02:31.935741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:31.935712 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:02:32.074030 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:02:32.073999 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7d0f9a_db83_4269_996e_20754563d353.slice/crio-85fef03a554a59132d997cc52a1d7c923fafd1992842703569f5dea56a1d1e4a WatchSource:0}: Error finding container 85fef03a554a59132d997cc52a1d7c923fafd1992842703569f5dea56a1d1e4a: Status 404 returned error can't find the container with id 85fef03a554a59132d997cc52a1d7c923fafd1992842703569f5dea56a1d1e4a Nov 04 03:02:32.074462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.074437 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:02:32.789610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.789570 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" event={"ID":"1a7d0f9a-db83-4269-996e-20754563d353","Type":"ContainerStarted","Data":"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132"} Nov 04 03:02:32.789850 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.789640 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" event={"ID":"1a7d0f9a-db83-4269-996e-20754563d353","Type":"ContainerStarted","Data":"85fef03a554a59132d997cc52a1d7c923fafd1992842703569f5dea56a1d1e4a"} Nov 04 03:02:32.789850 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.789823 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:32.794246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.793355 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" event={"ID":"a786813b-3dfa-405d-be13-45061bf3e640","Type":"ContainerStarted","Data":"4b304a731c69ecb37ebc21c3eb6d5ecbfc8bc02089ef2b5b37a5d0b37c302a4e"} Nov 04 03:02:32.794246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.793428 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" event={"ID":"a786813b-3dfa-405d-be13-45061bf3e640","Type":"ContainerStarted","Data":"412b8e5097a26ca0a45b754b278595d8fb1dc74fde09e70c844155afd3bc1fc8"} Nov 04 03:02:32.794450 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.794368 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:32.910006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:32.909913 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" podStartSLOduration=1.909890047 podStartE2EDuration="1.909890047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:32.907366609 +0000 UTC m=+1799.873867600" watchObservedRunningTime="2026-11-04 03:02:32.909890047 +0000 UTC m=+1799.876391040" Nov 04 03:02:33.031472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:33.031415 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" podStartSLOduration=3.031399241 podStartE2EDuration="3.031399241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:33.026115675 +0000 UTC m=+1799.992616669" watchObservedRunningTime="2026-11-04 03:02:33.031399241 +0000 UTC m=+1799.997900233" Nov 04 03:02:33.667798 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:33.667762 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:02:33.671260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:33.671232 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:02:33.673055 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:33.673031 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:02:33.676259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:33.676238 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:02:35.583388 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:35.583344 4784 scope.go:122] "RemoveContainer" containerID="099e1f4cef6285bf4b1170acaf531e2ae00d7bb865b4823a1efe792aceabb83f" Nov 04 03:02:35.613452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:35.613418 4784 scope.go:122] "RemoveContainer" containerID="d1ae222f06ec9b3e45b43b3a1d9dd2a93f20a4896fbb3022988185fe64bba2fb" Nov 04 03:02:35.624588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:35.624561 4784 scope.go:122] "RemoveContainer" containerID="3727f6823c1711bb0bf2628ec74809fca5bc79421bcc0a9c209e90e29591a6b7" Nov 04 03:02:35.635085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:35.635062 4784 scope.go:122] "RemoveContainer" containerID="79444fe222f1ce7f6df037a2205f231337680c26064ba479e62e161941aeef9d" Nov 04 03:02:36.423027 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:02:36.422978 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:02:43.077261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:43.075232 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4"] Nov 04 03:02:43.113510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:43.113463 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0969dd-hlcw4"] Nov 04 03:02:43.173255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:43.170241 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452ef2e9-e467-48d0-b381-3aaf5eabcbc4" path="/var/lib/kubelet/pods/452ef2e9-e467-48d0-b381-3aaf5eabcbc4/volumes" Nov 04 03:02:43.805437 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:43.805409 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:02:44.820463 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:44.820429 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:02:54.187362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:54.187328 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 03:02:54.187976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:54.187561 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" containerID="cri-o://f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81" gracePeriod=30 Nov 04 03:02:56.659278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.659244 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:02:56.664129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.664107 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.695005 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.694969 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696535 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696623 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn959\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696718 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696805 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696855 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.696973 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696906 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.697026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.696971 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.697080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.697072 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.718406 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.718374 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a021b3-openshift-default-dockercfg-ft9n2\"" Nov 04 03:02:56.776983 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.776944 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:02:56.826265 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.826224 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.826473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.826324 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.826473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.826395 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn959\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.826599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.826488 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.826599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.826587 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.829231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.829150 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.829651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.829627 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.829929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.829906 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.830433 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.830412 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.840996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.840946 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.854954 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.853488 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.864570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.835673 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.864570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.855549 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.864570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.830309 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.866620 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.866594 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.866945 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.866920 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.892240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.891719 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn959\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.918808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.918773 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\") pod \"e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:56.978627 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:56.978596 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:57.422579 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:02:57.422546 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b05cb5_f341_437e_bf0b_1f2ee9784cb2.slice/crio-958e019a156736b8dd947af37f237e2d226478c9fb5c27302ae4571de4ee9c14 WatchSource:0}: Error finding container 958e019a156736b8dd947af37f237e2d226478c9fb5c27302ae4571de4ee9c14: Status 404 returned error can't find the container with id 958e019a156736b8dd947af37f237e2d226478c9fb5c27302ae4571de4ee9c14 Nov 04 03:02:57.423099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:57.423074 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:02:57.425679 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:57.425645 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:57.425780 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:57.425712 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:57.425780 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:57.425744 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:02:58.157761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:58.157727 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerStarted","Data":"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9"} Nov 04 03:02:58.157761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:58.157775 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerStarted","Data":"958e019a156736b8dd947af37f237e2d226478c9fb5c27302ae4571de4ee9c14"} Nov 04 03:02:58.285955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:58.285897 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podStartSLOduration=2.285882233 podStartE2EDuration="2.285882233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:58.282320797 +0000 UTC m=+1825.248821788" watchObservedRunningTime="2026-11-04 03:02:58.285882233 +0000 UTC m=+1825.252383279" Nov 04 03:02:58.979284 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:58.979245 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:58.985339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:58.985315 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:59.166671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.166638 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:59.167712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.167694 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:02:59.520559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.520532 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:02:59.617980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.617897 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.617980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.617973 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618008 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618063 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618143 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618249 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:02:59.618398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618349 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618388 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618431 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618438 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:02:59.618555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618468 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\") pod \"efb0907e-7ded-4261-971b-6149ac28fb07\" (UID: \"efb0907e-7ded-4261-971b-6149ac28fb07\") " Nov 04 03:02:59.618555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618479 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:02:59.618829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618556 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:02:59.618829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.618765 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:02:59.619225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.619169 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.619225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.619208 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.619225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.619220 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.619225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.619228 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb0907e-7ded-4261-971b-6149ac28fb07-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.619225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.619239 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.620489 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.620463 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:02:59.620888 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.620864 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "kube-api-access-t7xb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:02:59.621150 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.621133 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:02:59.621236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.621169 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token" pod "efb0907e-7ded-4261-971b-6149ac28fb07" (UID: "efb0907e-7ded-4261-971b-6149ac28fb07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:02:59.721467 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.721430 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.721467 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.721468 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb0907e-7ded-4261-971b-6149ac28fb07-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.721724 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.721484 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xb7\" (UniqueName: \"kubernetes.io/projected/efb0907e-7ded-4261-971b-6149ac28fb07-kube-api-access-t7xb7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:02:59.721724 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:02:59.721499 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb0907e-7ded-4261-971b-6149ac28fb07-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:00.175965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.175933 4784 generic.go:356] "Generic (PLEG): container finished" podID="efb0907e-7ded-4261-971b-6149ac28fb07" containerID="f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81" exitCode=0 Nov 04 03:03:00.176472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.175987 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" Nov 04 03:03:00.176472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.176016 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerDied","Data":"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81"} Nov 04 03:03:00.176472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.176073 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj" event={"ID":"efb0907e-7ded-4261-971b-6149ac28fb07","Type":"ContainerDied","Data":"263c5751395e780db3a49018fab3b9cf015568b914ab2bca3a8c7a72f1d8ff86"} Nov 04 03:03:00.176472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.176096 4784 scope.go:122] "RemoveContainer" containerID="f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81" Nov 04 03:03:00.188547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.188520 4784 scope.go:122] "RemoveContainer" containerID="250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4" Nov 04 03:03:00.199600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.199579 4784 scope.go:122] "RemoveContainer" containerID="f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81" Nov 04 03:03:00.199868 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:03:00.199851 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81\": container with ID starting with f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81 not found: ID does not exist" containerID="f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81" Nov 04 03:03:00.199919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.199878 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81"} err="failed to get container status \"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81\": rpc error: code = NotFound desc = could not find container \"f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81\": container with ID starting with f8b20f0db03552f9210c179c1fe5881ff6d7ad2b0761d191b5ccbe3c070eda81 not found: ID does not exist" Nov 04 03:03:00.199919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.199898 4784 scope.go:122] "RemoveContainer" containerID="250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4" Nov 04 03:03:00.200133 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:03:00.200115 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4\": container with ID starting with 250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4 not found: ID does not exist" containerID="250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4" Nov 04 03:03:00.200178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.200139 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4"} err="failed to get container status \"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4\": rpc error: code = NotFound desc = could not find container \"250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4\": container with ID starting with 250c5a3fd206998e4a07d2dab09d07314617c5decb5c9229f48e086787325ad4 not found: ID does not exist" Nov 04 03:03:00.399456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.399421 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 03:03:00.450600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:00.450519 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-db4d0787-openshift-default-7bd494bcfc-zqfcj"] Nov 04 03:03:01.171504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:01.171472 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" path="/var/lib/kubelet/pods/efb0907e-7ded-4261-971b-6149ac28fb07/volumes" Nov 04 03:03:06.402074 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:03:06.402028 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:03:16.484141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.484057 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h"] Nov 04 03:03:16.484905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.484889 4784 cpu_manager.go:414] "containerMap: removing container" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.484955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.484907 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.484955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.484922 4784 cpu_manager.go:414] "containerMap: removing container" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.484955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.484928 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.485060 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.485051 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.485096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.485062 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb0907e-7ded-4261-971b-6149ac28fb07" containerName="istio-proxy" Nov 04 03:03:16.490529 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.490508 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.557988 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.557951 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h"] Nov 04 03:03:16.603242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.603156 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:03:16.603242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.603181 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:03:16.603530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.603267 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:03:16.603530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.603270 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:03:16.621697 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.621666 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4b8\" (UniqueName: \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.621959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.621939 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.622115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.622094 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.723807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.723767 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.723989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.723915 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.724032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.723985 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4b8\" (UniqueName: \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.724548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.724519 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.726762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.726733 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.786771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.786692 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4b8\" (UniqueName: \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\") pod \"maas-api-revoke-sub-95359c5c-dr78h\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:16.805748 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:16.805714 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:17.200527 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:03:17.200490 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba05c3c_d631_4117_a835_ca11ea6a9acd.slice/crio-8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4 WatchSource:0}: Error finding container 8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4: Status 404 returned error can't find the container with id 8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4 Nov 04 03:03:17.200696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:17.200673 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h"] Nov 04 03:03:17.332971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:17.332928 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" event={"ID":"aba05c3c-d631-4117-a835-ca11ea6a9acd","Type":"ContainerStarted","Data":"8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4"} Nov 04 03:03:18.345467 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:18.345432 4784 generic.go:356] "Generic (PLEG): container finished" podID="aba05c3c-d631-4117-a835-ca11ea6a9acd" containerID="4a7b9921b0129fd9ad44b287fd5b0cdfc34e39b1cd5f9423dfd766d42af04b0d" exitCode=0 Nov 04 03:03:18.345860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:18.345470 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" event={"ID":"aba05c3c-d631-4117-a835-ca11ea6a9acd","Type":"ContainerDied","Data":"4a7b9921b0129fd9ad44b287fd5b0cdfc34e39b1cd5f9423dfd766d42af04b0d"} Nov 04 03:03:19.527861 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.527831 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:19.672419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.672374 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\") pod \"aba05c3c-d631-4117-a835-ca11ea6a9acd\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " Nov 04 03:03:19.672609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.672491 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\") pod \"aba05c3c-d631-4117-a835-ca11ea6a9acd\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " Nov 04 03:03:19.672609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.672526 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\" (UniqueName: \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\") pod \"aba05c3c-d631-4117-a835-ca11ea6a9acd\" (UID: \"aba05c3c-d631-4117-a835-ca11ea6a9acd\") " Nov 04 03:03:19.672817 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.672790 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca" pod "aba05c3c-d631-4117-a835-ca11ea6a9acd" (UID: "aba05c3c-d631-4117-a835-ca11ea6a9acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:03:19.673461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.673434 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:19.674989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.674962 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca" pod "aba05c3c-d631-4117-a835-ca11ea6a9acd" (UID: "aba05c3c-d631-4117-a835-ca11ea6a9acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:03:19.674989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.674962 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8" pod "aba05c3c-d631-4117-a835-ca11ea6a9acd" (UID: "aba05c3c-d631-4117-a835-ca11ea6a9acd"). InnerVolumeSpecName "kube-api-access-vb4b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:03:19.775074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.775037 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba05c3c-d631-4117-a835-ca11ea6a9acd-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:19.775074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:19.775069 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4b8\" (UniqueName: \"kubernetes.io/projected/aba05c3c-d631-4117-a835-ca11ea6a9acd-kube-api-access-vb4b8\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:20.363052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:20.363021 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" Nov 04 03:03:20.363299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:20.363057 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h" event={"ID":"aba05c3c-d631-4117-a835-ca11ea6a9acd","Type":"ContainerDied","Data":"8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4"} Nov 04 03:03:20.363299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:20.363113 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8910db7a6a9a1f0c9babd61e7191431231ba37971ccefb294bfad72a7e9771e4" Nov 04 03:03:35.712136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:35.712102 4784 scope.go:122] "RemoveContainer" containerID="bb4e3b95aa5a1531ca3199012be2f689f0ab6d802c0a670a12126e752b23ae68" Nov 04 03:03:36.402299 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:03:36.402260 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:03:36.533522 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.533490 4784 generic.go:356] "Generic (PLEG): container finished" podID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerID="be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd" exitCode=0 Nov 04 03:03:36.533699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.533564 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerDied","Data":"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd"} Nov 04 03:03:36.534059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.534043 4784 scope.go:122] "RemoveContainer" containerID="be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd" Nov 04 03:03:36.535807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.535777 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:36.535892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.535847 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:36.535892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:36.535885 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:37.544737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:37.544704 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerStarted","Data":"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a"} Nov 04 03:03:38.215070 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.215033 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 03:03:38.215290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.215262 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" containerID="cri-o://e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596" gracePeriod=30 Nov 04 03:03:38.223987 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.223950 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:03:38.225716 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.225689 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:03:38.225822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.225742 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:03:38.316873 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:38.316843 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:03:39.224166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:39.224131 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:03:39.224636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:39.224182 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:03:40.209280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.209244 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:03:40.210007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.209994 4784 cpu_manager.go:414] "containerMap: removing container" podUID="aba05c3c-d631-4117-a835-ca11ea6a9acd" containerName="revoke-keys" Nov 04 03:03:40.210048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.210009 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba05c3c-d631-4117-a835-ca11ea6a9acd" containerName="revoke-keys" Nov 04 03:03:40.210126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.210116 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba05c3c-d631-4117-a835-ca11ea6a9acd" containerName="revoke-keys" Nov 04 03:03:40.212860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.212842 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.223741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.223713 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:03:40.224053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.223773 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:03:40.296761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.296725 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:03:40.315617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.315583 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.315917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.315893 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.316092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.316073 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffrj\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.316353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.316319 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.316556 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.316542 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.316759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.316742 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.316946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.316930 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.317129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.317113 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.317495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.317474 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.324951 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.324928 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cdb11d-openshift-default-dockercfg-f28m2\"" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.427956 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428299 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428350 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428394 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428421 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffrj\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428500 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428543 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428635 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.428697 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.429627 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.435214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.429916 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.437887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.437857 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.438361 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.438337 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.438927 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.438876 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.453269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.452549 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.453269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.452983 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.500892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.500813 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.501046 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.501014 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffrj\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\") pod \"e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:40.526818 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:40.526783 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:41.081254 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:03:41.055467 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35dceb6_82b8_49ab_8983_d4fbf58d8fe9.slice/crio-f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990 WatchSource:0}: Error finding container f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990: Status 404 returned error can't find the container with id f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990 Nov 04 03:03:41.081254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.074298 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:03:41.082824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.082780 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:41.082962 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.082878 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:41.082962 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.082910 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:41.120297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.120261 4784 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:03:41.120445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.120328 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:03:41.229230 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.223692 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:03:41.229230 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.223757 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:03:41.613466 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.613427 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerStarted","Data":"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c"} Nov 04 03:03:41.613938 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.613492 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerStarted","Data":"f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990"} Nov 04 03:03:41.741127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:41.741063 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podStartSLOduration=1.741044208 podStartE2EDuration="1.741044208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:03:41.736412078 +0000 UTC m=+1868.702913073" watchObservedRunningTime="2026-11-04 03:03:41.741044208 +0000 UTC m=+1868.707545202" Nov 04 03:03:42.223956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.223920 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 03:03:42.224144 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.223981 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 03:03:42.534272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.527442 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:42.538296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.538007 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:42.622712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.622676 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:42.623658 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:42.623632 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:03:43.228983 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.228948 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:03:43.230003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.229978 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:03:43.517152 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.510653 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 03:03:43.617648 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.617614 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.617836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.617708 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.617836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.617732 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.617951 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.617892 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.617951 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.617940 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.618074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618032 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:43.618136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618087 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:43.618136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618097 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.618257 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618173 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:43.618445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618420 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:03:43.618584 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618566 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.618716 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618701 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.618778 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618732 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\") pod \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\" (UID: \"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925\") " Nov 04 03:03:43.619008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.618987 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:43.620212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.619904 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.620212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.619934 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.620212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.619950 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.620212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.619964 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.620212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.619978 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.620663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.620541 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:43.620728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.620671 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:03:43.621148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.621123 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:03:43.621270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.621237 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp" pod "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" (UID: "0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925"). InnerVolumeSpecName "kube-api-access-cvrjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:03:43.635225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.633343 4784 generic.go:356] "Generic (PLEG): container finished" podID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerID="e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596" exitCode=0 Nov 04 03:03:43.635225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.633425 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" Nov 04 03:03:43.635225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.633495 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" event={"ID":"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925","Type":"ContainerDied","Data":"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596"} Nov 04 03:03:43.635225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.633539 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd" event={"ID":"0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925","Type":"ContainerDied","Data":"d77bf54d85ae1e374e45cfa57bf67b54cc88deaa811c5acce2e0faf626efab3a"} Nov 04 03:03:43.635225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.633565 4784 scope.go:122] "RemoveContainer" containerID="e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596" Nov 04 03:03:43.668614 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.668587 4784 scope.go:122] "RemoveContainer" containerID="e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596" Nov 04 03:03:43.668967 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:03:43.668948 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596\": container with ID starting with e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596 not found: ID does not exist" containerID="e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596" Nov 04 03:03:43.669089 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.668975 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596"} err="failed to get container status \"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596\": rpc error: code = NotFound desc = could not find container \"e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596\": container with ID starting with e5e1acde39054241af8496176115e3c0513911dfb557d1eba0a6c3578738e596 not found: ID does not exist" Nov 04 03:03:43.722568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.722503 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.722568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.722567 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.722839 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.722586 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvrjp\" (UniqueName: \"kubernetes.io/projected/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-kube-api-access-cvrjp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.722839 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.722599 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:43.815274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.815132 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 03:03:43.819823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:43.819789 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-7mhsd"] Nov 04 03:03:44.059853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:44.059817 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7"] Nov 04 03:03:44.065359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:44.065266 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c5d4d3-kmrk7"] Nov 04 03:03:45.162869 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:45.162833 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" path="/var/lib/kubelet/pods/0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925/volumes" Nov 04 03:03:45.163362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:45.163331 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f24919a-96b5-42fd-8109-0b59e34905ef" path="/var/lib/kubelet/pods/9f24919a-96b5-42fd-8109-0b59e34905ef/volumes" Nov 04 03:03:58.924214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:58.923884 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:03:58.948212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:58.947152 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" Nov 04 03:03:58.948212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:58.947201 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" Nov 04 03:03:58.948212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:58.947500 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2bd098-f7bb-4d5c-9c95-1c7dfe73e925" containerName="istio-proxy" Nov 04 03:03:58.960863 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:58.960569 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.030052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.030000 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:03:59.070214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.069420 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-00c846-openshift-default-dockercfg-sd6xs\"" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.070477 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.070572 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.070686 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.070750 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.071002 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.071051 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.071089 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.071166 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.080221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.071259 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpc7\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.172849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.172817 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.172849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.172864 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.172895 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.172944 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.172986 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpc7\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173029 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173098 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173409 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173248 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173633 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173609 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.173777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173752 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.174078 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.174056 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.174295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.174240 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.174295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.174272 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.174430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.173307 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.176013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.175994 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.177076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.177054 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.250044 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.250008 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.256087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.256008 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpc7\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\") pod \"e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.300517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.300476 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:03:59.830379 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:03:59.830349 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ebf2e9_1003_4ae4_8710_bcabe3592dd0.slice/crio-371bf23f8e8aa5893c6f296045ec5fc0e49bacf98b49ce6572e609917de4fc0f WatchSource:0}: Error finding container 371bf23f8e8aa5893c6f296045ec5fc0e49bacf98b49ce6572e609917de4fc0f: Status 404 returned error can't find the container with id 371bf23f8e8aa5893c6f296045ec5fc0e49bacf98b49ce6572e609917de4fc0f Nov 04 03:03:59.833236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.833211 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:03:59.833600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.833568 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:59.833643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.833633 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:59.833682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.833663 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:03:59.954347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.954298 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerStarted","Data":"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0"} Nov 04 03:03:59.954816 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:03:59.954370 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerStarted","Data":"371bf23f8e8aa5893c6f296045ec5fc0e49bacf98b49ce6572e609917de4fc0f"} Nov 04 03:04:00.087250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:00.087125 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podStartSLOduration=2.087110915 podStartE2EDuration="2.087110915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:00.085404588 +0000 UTC m=+1887.051905579" watchObservedRunningTime="2026-11-04 03:04:00.087110915 +0000 UTC m=+1887.053611906" Nov 04 03:04:00.301049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:00.301017 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:04:00.302918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:00.302890 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:04:00.303067 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:00.302976 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:04:01.306513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:01.306481 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:04:01.972784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:01.972750 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:04:01.973736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:01.973714 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:04:06.413279 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:04:06.413218 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:04:17.806607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:17.806568 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:04:17.812721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:17.812694 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:17.867418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:17.867367 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-675c80-openshift-default-dockercfg-fgfv4\"" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017429 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017583 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017701 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017768 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.018555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.017835 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.025374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.025343 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:04:18.104702 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.104588 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.105084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.105057 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.105446 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.105424 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq4t\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.207644 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207608 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.207831 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207659 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.207831 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207698 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq4t\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.207831 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207813 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207896 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207949 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.207990 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208023 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208053 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208119 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208780 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208785 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.208906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.208813 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.209301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.209273 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.210805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.210783 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.210896 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.210837 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.271107 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.271072 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.271418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.271253 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq4t\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\") pod \"e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.430228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.428389 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:18.973930 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:04:18.973902 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12d7995_7b2a_406c_9c56_e0a899176738.slice/crio-03da50c09055dcd241fcc1ea6369922f3660ee8d4338eb0bb9bb3d8383f5a220 WatchSource:0}: Error finding container 03da50c09055dcd241fcc1ea6369922f3660ee8d4338eb0bb9bb3d8383f5a220: Status 404 returned error can't find the container with id 03da50c09055dcd241fcc1ea6369922f3660ee8d4338eb0bb9bb3d8383f5a220 Nov 04 03:04:18.974409 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.974384 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:04:18.977500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.977452 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:18.977789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.977536 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:18.977789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:18.977570 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:19.164815 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:19.164776 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerStarted","Data":"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa"} Nov 04 03:04:19.165021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:19.164829 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerStarted","Data":"03da50c09055dcd241fcc1ea6369922f3660ee8d4338eb0bb9bb3d8383f5a220"} Nov 04 03:04:19.285052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:19.284933 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podStartSLOduration=2.284910699 podStartE2EDuration="2.284910699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:19.282934753 +0000 UTC m=+1906.249435746" watchObservedRunningTime="2026-11-04 03:04:19.284910699 +0000 UTC m=+1906.251411691" Nov 04 03:04:19.441301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:19.441259 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:20.442383 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:20.442350 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:21.184845 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:21.184803 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:21.186003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:21.185979 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:04:28.387213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.383984 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:04:28.467028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.466988 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.478713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.478672 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.484269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.484233 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdbmr\" (UniqueName: \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.506079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.506041 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:04:28.572843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.572812 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-675c80\"" Nov 04 03:04:28.572843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.572812 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-675c80-dockercfg-hkd97\"" Nov 04 03:04:28.587250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.587136 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.587250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.587207 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdbmr\" (UniqueName: \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.588052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.588024 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.649783 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.649710 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdbmr\" (UniqueName: \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\") pod \"payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:28.823664 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:28.823611 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:29.224821 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:04:29.224776 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b158c0_ea09_413b_a88a_3de8f11a3565.slice/crio-94b2fb92d7c9d93cdd61054ced04456dbc1420fde176b893383c62181c9f9573 WatchSource:0}: Error finding container 94b2fb92d7c9d93cdd61054ced04456dbc1420fde176b893383c62181c9f9573: Status 404 returned error can't find the container with id 94b2fb92d7c9d93cdd61054ced04456dbc1420fde176b893383c62181c9f9573 Nov 04 03:04:29.225956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:29.225931 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:04:29.284262 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:29.284221 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" event={"ID":"e5b158c0-ea09-413b-a88a-3de8f11a3565","Type":"ContainerStarted","Data":"94b2fb92d7c9d93cdd61054ced04456dbc1420fde176b893383c62181c9f9573"} Nov 04 03:04:30.300237 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:30.300155 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" event={"ID":"e5b158c0-ea09-413b-a88a-3de8f11a3565","Type":"ContainerStarted","Data":"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2"} Nov 04 03:04:30.429116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:30.429047 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" podStartSLOduration=2.429024233 podStartE2EDuration="2.429024233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:30.42186344 +0000 UTC m=+1917.388364433" watchObservedRunningTime="2026-11-04 03:04:30.429024233 +0000 UTC m=+1917.395525228" Nov 04 03:04:31.311229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:31.311198 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:32.321939 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.321849 4784 generic.go:356] "Generic (PLEG): container finished" podID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerID="dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f" exitCode=0 Nov 04 03:04:32.322520 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.321921 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerDied","Data":"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f"} Nov 04 03:04:32.322641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.322622 4784 scope.go:122] "RemoveContainer" containerID="dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f" Nov 04 03:04:32.324288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.324261 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:32.324360 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.324329 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:32.324398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:32.324360 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:04:33.332857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:33.332821 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerStarted","Data":"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5"} Nov 04 03:04:33.513507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:33.513473 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:04:33.514552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:33.514519 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:33.514681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:33.514580 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:34.513614 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:34.513578 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:34.514135 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:34.513653 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:35.513169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:35.513129 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:35.513391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:35.513225 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:35.797795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:35.797596 4784 scope.go:122] "RemoveContainer" containerID="4903b4d910e34edd98d6f9c462c2f6427a552ac5203596799e79cd28b44ab452" Nov 04 03:04:35.811533 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:35.811509 4784 scope.go:122] "RemoveContainer" containerID="9ce57f40cfb385248840701569b11dd9881b4ff1f76acdb6974ea65333491210" Nov 04 03:04:35.822913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:35.822887 4784 scope.go:122] "RemoveContainer" containerID="966bb8f0202ba08e4f61b571e776ec7749b5448fed1fc9da0865e7752d6e2d79" Nov 04 03:04:36.416701 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:04:36.416658 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:04:36.513767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:36.513728 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:36.513961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:36.513795 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:37.515893 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:37.515858 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:37.516609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:37.516558 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:37.548058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:37.548028 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:04:38.513526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:38.513486 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 03:04:38.513727 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:38.513545 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 03:04:38.941229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:38.941135 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 03:04:38.941805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:38.941446 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" containerID="cri-o://859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f" gracePeriod=30 Nov 04 03:04:39.518479 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:39.518444 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:04:39.519787 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:39.519763 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:04:39.884048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:39.883965 4784 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:04:39.884048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:39.884029 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:04:42.326857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:42.326761 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:04:44.237810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.237784 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 03:04:44.310312 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310276 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310521 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310336 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310521 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310387 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310521 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310595 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:04:44.310777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310667 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310698 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310704 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:04:44.310777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310733 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.310928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310769 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:04:44.310928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.310870 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.311048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311010 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\") pod \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\" (UID: \"50fdd56c-88dd-4a2e-9a39-41be0777a0bb\") " Nov 04 03:04:44.311122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311047 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311358 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311831 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311855 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311869 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311884 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.311922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.311898 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.313009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.312988 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:04:44.313101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.313006 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:04:44.313101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.313054 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:04:44.313381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.313357 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82" pod "50fdd56c-88dd-4a2e-9a39-41be0777a0bb" (UID: "50fdd56c-88dd-4a2e-9a39-41be0777a0bb"). InnerVolumeSpecName "kube-api-access-mmt82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:04:44.414393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.414352 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.414393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.414390 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.414624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.414405 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmt82\" (UniqueName: \"kubernetes.io/projected/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-kube-api-access-mmt82\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.414624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.414419 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fdd56c-88dd-4a2e-9a39-41be0777a0bb-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:04:44.518623 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.518527 4784 generic.go:356] "Generic (PLEG): container finished" podID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerID="859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f" exitCode=0 Nov 04 03:04:44.518623 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.518579 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" Nov 04 03:04:44.518867 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.518618 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" event={"ID":"50fdd56c-88dd-4a2e-9a39-41be0777a0bb","Type":"ContainerDied","Data":"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f"} Nov 04 03:04:44.518867 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.518684 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95" event={"ID":"50fdd56c-88dd-4a2e-9a39-41be0777a0bb","Type":"ContainerDied","Data":"d48c7c28fec3f6071183fbbfc2a995a8ffc4c1605f8dfd8b90a4804f7308ed00"} Nov 04 03:04:44.518867 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.518713 4784 scope.go:122] "RemoveContainer" containerID="859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f" Nov 04 03:04:44.540410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.540391 4784 scope.go:122] "RemoveContainer" containerID="859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f" Nov 04 03:04:44.540733 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:04:44.540713 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f\": container with ID starting with 859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f not found: ID does not exist" containerID="859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f" Nov 04 03:04:44.540795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.540741 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f"} err="failed to get container status \"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f\": rpc error: code = NotFound desc = could not find container \"859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f\": container with ID starting with 859e6e0e14c240b74740b73e3d6e10eb855a2c0065923a726ffb054ce6a96f1f not found: ID does not exist" Nov 04 03:04:44.691329 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.691285 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 03:04:44.697905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:44.697862 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-hht95"] Nov 04 03:04:45.162680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:04:45.162648 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" path="/var/lib/kubelet/pods/50fdd56c-88dd-4a2e-9a39-41be0777a0bb/volumes" Nov 04 03:05:06.403728 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:05:06.403680 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:05:10.790373 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.790313 4784 generic.go:356] "Generic (PLEG): container finished" podID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerID="3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9" exitCode=0 Nov 04 03:05:10.790953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.790639 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerDied","Data":"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9"} Nov 04 03:05:10.791477 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.791453 4784 scope.go:122] "RemoveContainer" containerID="3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9" Nov 04 03:05:10.800095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.800033 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:10.800309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.800224 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:10.800373 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:10.800320 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:11.802068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:11.802025 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerStarted","Data":"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129"} Nov 04 03:05:11.980216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:11.980139 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:05:11.980410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:11.980343 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:11.980503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:11.980405 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:12.981687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:12.981651 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:12.982173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:12.981712 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:13.981372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:13.981329 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:13.981587 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:13.981394 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:14.173681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:14.173648 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:05:14.981658 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:14.981614 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:14.981989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:14.981958 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:15.090034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.089993 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:05:15.100127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.100097 4784 cpu_manager.go:414] "containerMap: removing container" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" Nov 04 03:05:15.100423 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.100356 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" Nov 04 03:05:15.100982 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.100812 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fdd56c-88dd-4a2e-9a39-41be0777a0bb" containerName="istio-proxy" Nov 04 03:05:15.145214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.119253 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.176204 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.176153 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9d2b96-openshift-default-dockercfg-gvxzv\"" Nov 04 03:05:15.326612 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.267322 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.326878 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.326854 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.327038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.327019 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.327165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.327152 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.335408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.335376 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.382477 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.382435 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.382764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.382738 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.382991 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.382966 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jr2\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.383438 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.383409 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.514936 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.500752 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.516066 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.516030 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.516926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.516903 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.517514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.517484 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.518376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.518354 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.519426 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.516796 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.519584 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.510181 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.519719 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.518797 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.519814 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.519466 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.519925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.519913 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.520016 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.520003 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jr2\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.520227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.520205 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.521409 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.519730 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.521519 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.518219 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.535323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.535240 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:05:15.535947 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.535919 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.537502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.537476 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.679302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.632110 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.745683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.745625 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jr2\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\") pod \"e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.779259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.770456 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:15.991387 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.984326 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:15.991387 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:15.984385 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:16.380391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.379016 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:05:16.387347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.387296 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:16.387512 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.387403 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:16.387512 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.387450 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:16.895690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.895642 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerStarted","Data":"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c"} Nov 04 03:05:16.895910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.895719 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerStarted","Data":"1485598c5952e42d5ef709d5b54805372b263d138d5602436049cf89e3d314d7"} Nov 04 03:05:16.982214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.981532 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 04 03:05:16.982214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:16.981591 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 04 03:05:17.019620 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.019568 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podStartSLOduration=2.019552571 podStartE2EDuration="2.019552571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:05:17.018239323 +0000 UTC m=+1963.984740316" watchObservedRunningTime="2026-11-04 03:05:17.019552571 +0000 UTC m=+1963.986053575" Nov 04 03:05:17.771671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.771636 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:17.777492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.777467 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:17.907053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.907017 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:17.908243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.908222 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:05:17.999217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:17.998616 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:05:18.011414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:18.010426 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:05:27.109453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.109410 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:05:27.114648 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.114623 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.175546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.175514 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:05:27.231323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.215703 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9d2b96-dockercfg-2w27p\"" Nov 04 03:05:27.231323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.215842 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9d2b96\"" Nov 04 03:05:27.231323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.224660 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.231323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.224993 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqkc\" (UniqueName: \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.327692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.327653 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqkc\" (UniqueName: \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.328141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.328119 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.329596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.329573 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.460655 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.460046 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqkc\" (UniqueName: \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\") pod \"payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:27.741939 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:27.741851 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:28.125532 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:05:28.125497 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4345f149_42a6_4102_9412_3353b0ac2837.slice/crio-6f80d39d8df5adf18ae138ae409f37c0eb7dfaaf6637d004c6df515954b9ce97 WatchSource:0}: Error finding container 6f80d39d8df5adf18ae138ae409f37c0eb7dfaaf6637d004c6df515954b9ce97: Status 404 returned error can't find the container with id 6f80d39d8df5adf18ae138ae409f37c0eb7dfaaf6637d004c6df515954b9ce97 Nov 04 03:05:28.126077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:28.125980 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:05:29.099094 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:29.099053 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" event={"ID":"4345f149-42a6-4102-9412-3353b0ac2837","Type":"ContainerStarted","Data":"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f"} Nov 04 03:05:29.099340 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:29.099111 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" event={"ID":"4345f149-42a6-4102-9412-3353b0ac2837","Type":"ContainerStarted","Data":"6f80d39d8df5adf18ae138ae409f37c0eb7dfaaf6637d004c6df515954b9ce97"} Nov 04 03:05:29.099340 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:29.099133 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:29.220883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:29.220821 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" podStartSLOduration=2.22080559 podStartE2EDuration="2.22080559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:05:29.216849954 +0000 UTC m=+1976.183350950" watchObservedRunningTime="2026-11-04 03:05:29.22080559 +0000 UTC m=+1976.187306582" Nov 04 03:05:35.910647 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:35.910613 4784 scope.go:122] "RemoveContainer" containerID="e34625270b92865ecd3802aff46e11d4bc5d49a0d2677177568970aab80713b3" Nov 04 03:05:36.402933 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:05:36.402889 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:05:40.111550 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:40.111520 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:05:58.183784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.183170 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": read tcp 10.133.0.2:49028->10.133.0.194:15021: read: connection reset by peer" start-of-body= Nov 04 03:05:58.183784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.183356 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": read tcp 10.133.0.2:49028->10.133.0.194:15021: read: connection reset by peer" Nov 04 03:05:58.461470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.461382 4784 generic.go:356] "Generic (PLEG): container finished" podID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerID="5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c" exitCode=0 Nov 04 03:05:58.461627 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.461460 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerDied","Data":"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c"} Nov 04 03:05:58.461980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.461966 4784 scope.go:122] "RemoveContainer" containerID="5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c" Nov 04 03:05:58.463967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.463935 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:58.464054 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.464015 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:58.464091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:58.464058 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:05:59.488661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:59.488625 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerStarted","Data":"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b"} Nov 04 03:05:59.527592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:59.527546 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:05:59.527839 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:59.527818 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:05:59.527916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:05:59.527864 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:00.527593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:00.527554 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:06:00.528023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:00.527612 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:01.527478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:01.527438 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:06:01.527680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:01.527510 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:02.527589 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:02.527551 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:06:02.527802 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:02.527606 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:03.528140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:03.528107 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:06:03.528684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:03.528169 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:04.527202 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:04.527144 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 04 03:06:04.527547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:04.527223 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 04 03:06:05.532511 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:05.532479 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:06:05.533012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:05.532990 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:06:05.533872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:05.533850 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:06:06.402918 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:06.402883 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:06:14.792459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:14.792370 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:06:14.792946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:14.792595 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" containerID="cri-o://32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129" gracePeriod=30 Nov 04 03:06:17.748047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.748011 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:06:17.752553 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.752530 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.822686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.822654 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9f4199-openshift-default-dockercfg-7nr2f\"" Nov 04 03:06:17.835485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.835442 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.835897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.835872 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.836132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.836114 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.836333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.836303 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.836559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.836536 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.836916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.836665 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.837215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.837172 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.837402 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.837384 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.837541 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.837524 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5q9\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960512 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960561 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960702 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960769 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960813 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5q9\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.960964 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.961039 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.961111 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.961149 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.961946 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.964362 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.967908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.965019 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.995231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.990416 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:17.995231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:17.990837 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.062537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.061217 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5q9\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.085222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.083459 4784 patch_prober.go:29] interesting pod/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:18.085222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.083519 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:18.085524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.085492 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.114243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.113566 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.114243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.113869 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\") pod \"e2e-inf-a-9f4199-openshift-default-586448895-hl8h4\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.115148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.115092 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:06:18.148998 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.148966 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:18.684582 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:06:18.684550 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afd3c41_3375_4ae7_8e07_e18676b0c3ec.slice/crio-a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62 WatchSource:0}: Error finding container a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62: Status 404 returned error can't find the container with id a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62 Nov 04 03:06:18.687781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.687744 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:18.687874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.687817 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:18.687874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.687846 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:18.688105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.688083 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:06:18.731202 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:18.731151 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerStarted","Data":"a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62"} Nov 04 03:06:19.742548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:19.742508 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerStarted","Data":"ccc80c34705355278a817cc9f4c77051f872faf20d0c6592b0cf621165ac004a"} Nov 04 03:06:19.883271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:19.882158 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podStartSLOduration=2.882137316 podStartE2EDuration="2.882137316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:19.878684414 +0000 UTC m=+2026.845185408" watchObservedRunningTime="2026-11-04 03:06:19.882137316 +0000 UTC m=+2026.848638309" Nov 04 03:06:20.149490 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.149456 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:20.155464 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.155434 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:20.229769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.229740 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:06:20.352043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.351955 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352037 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352113 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352143 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352163 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352226 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352289 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352324 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352385 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\") pod \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\" (UID: \"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2\") " Nov 04 03:06:20.352820 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352768 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:20.352969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352860 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:20.352969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.352938 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.353099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.353019 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:20.353099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.353051 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:20.353250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.353212 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:20.375214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.370464 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:20.386405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.386366 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:20.386405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.386401 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "kube-api-access-tn959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:20.386603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.386370 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token" pod "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" (UID: "a3b05cb5-f341-437e-bf0b-1f2ee9784cb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:20.454768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454722 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.454768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454759 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.454768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454772 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.455058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454785 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn959\" (UniqueName: \"kubernetes.io/projected/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-kube-api-access-tn959\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.455058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454799 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.455058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454811 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.455058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454822 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.455058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.454835 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:20.741114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.741079 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:06:20.742047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742026 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.742108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742049 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.742108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742067 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.742108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742072 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.742253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742204 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.742253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.742221 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerName="istio-proxy" Nov 04 03:06:20.748356 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.748330 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.763085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.755676 4784 generic.go:356] "Generic (PLEG): container finished" podID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" containerID="32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129" exitCode=0 Nov 04 03:06:20.777227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.774862 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" Nov 04 03:06:20.782208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.777439 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerDied","Data":"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129"} Nov 04 03:06:20.782208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.777569 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:20.782208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.777622 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw" event={"ID":"a3b05cb5-f341-437e-bf0b-1f2ee9784cb2","Type":"ContainerDied","Data":"958e019a156736b8dd947af37f237e2d226478c9fb5c27302ae4571de4ee9c14"} Nov 04 03:06:20.782208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.777657 4784 scope.go:122] "RemoveContainer" containerID="32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129" Nov 04 03:06:20.790072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.787833 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:06:20.793120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.792353 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:06:20.793120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.792405 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.802564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.802538 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d2b8ee-openshift-default-dockercfg-bd27j\"" Nov 04 03:06:20.808709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.808682 4784 scope.go:122] "RemoveContainer" containerID="3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9" Nov 04 03:06:20.821021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.820999 4784 scope.go:122] "RemoveContainer" containerID="32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129" Nov 04 03:06:20.821449 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:20.821337 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129\": container with ID starting with 32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129 not found: ID does not exist" containerID="32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129" Nov 04 03:06:20.821449 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.821368 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129"} err="failed to get container status \"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129\": rpc error: code = NotFound desc = could not find container \"32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129\": container with ID starting with 32fdbf7ea4bafbdc2a2904a251c26a6dddbbfbbe8636acd470fad07d02505129 not found: ID does not exist" Nov 04 03:06:20.821449 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.821387 4784 scope.go:122] "RemoveContainer" containerID="3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9" Nov 04 03:06:20.821712 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:20.821666 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9\": container with ID starting with 3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9 not found: ID does not exist" containerID="3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9" Nov 04 03:06:20.821712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.821686 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9"} err="failed to get container status \"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9\": rpc error: code = NotFound desc = could not find container \"3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9\": container with ID starting with 3fc844860c55180bcb7fe0b5d62b7e41cec906d1f8220abc9fe4f1ef562f12d9 not found: ID does not exist" Nov 04 03:06:20.851620 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.851585 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:06:20.887499 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887453 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.887689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887543 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.887689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887576 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.887689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887617 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.887689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887670 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgl5\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.887837 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887716 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp468\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.887837 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887757 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.887837 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887831 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.887924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887872 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.887968 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888294 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888365 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888387 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888589 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888610 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888637 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888683 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.889140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.888711 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.907751 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.907704 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:06:20.990535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990415 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgl5\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.990535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990467 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp468\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.990535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990515 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.990806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990585 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.990806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990625 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.990806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990704 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.990806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990759 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.990937 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990802 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.990937 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.990831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991007 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.991086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991039 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991081 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991146 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.991347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991208 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.991347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991262 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991309 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991347 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991393 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.991510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991448 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991481 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.991719 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.991512 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.992172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992052 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.992172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992064 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.992172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992123 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.992481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992407 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.992609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992491 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.992791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.992772 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.993112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.993090 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.993731 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.993703 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.995173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.995140 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:20.996001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.995979 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:20.996571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:20.996552 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:21.086909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.086881 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgl5\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:21.087151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.086951 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:21.087334 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.087164 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp468\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:21.087619 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.087597 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:21.095234 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.095205 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:21.116166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.116143 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:21.142276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.142239 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:06:21.151804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.151740 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a021b3-openshift-default-6476c6cc55-6zjzw"] Nov 04 03:06:21.246745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.246665 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b05cb5-f341-437e-bf0b-1f2ee9784cb2" path="/var/lib/kubelet/pods/a3b05cb5-f341-437e-bf0b-1f2ee9784cb2/volumes" Nov 04 03:06:21.577015 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.576900 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:06:21.577378 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:06:21.577349 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45024334_79f6_4211_bdf2_9c5f4106dd59.slice/crio-44f84c34eaed19a05e06cd28b4c38fc4168f8fcdbaf4f4f97e2f3ee2fde5128a WatchSource:0}: Error finding container 44f84c34eaed19a05e06cd28b4c38fc4168f8fcdbaf4f4f97e2f3ee2fde5128a: Status 404 returned error can't find the container with id 44f84c34eaed19a05e06cd28b4c38fc4168f8fcdbaf4f4f97e2f3ee2fde5128a Nov 04 03:06:21.580627 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:06:21.580591 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd9f487_686c_4630_a511_03599a0e831e.slice/crio-ce1eb114c424a13971c8c48b08d53dece672dee3c3c3a6cc16634cd06646f1cf WatchSource:0}: Error finding container ce1eb114c424a13971c8c48b08d53dece672dee3c3c3a6cc16634cd06646f1cf: Status 404 returned error can't find the container with id ce1eb114c424a13971c8c48b08d53dece672dee3c3c3a6cc16634cd06646f1cf Nov 04 03:06:21.580773 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.580727 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.580855 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.580833 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.580923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.580886 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.584269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.584230 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.584378 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.584333 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.584452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.584379 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:21.584607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.584335 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:06:21.776309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.776279 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" event={"ID":"45024334-79f6-4211-bdf2-9c5f4106dd59","Type":"ContainerStarted","Data":"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c"} Nov 04 03:06:21.776794 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.776763 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" event={"ID":"45024334-79f6-4211-bdf2-9c5f4106dd59","Type":"ContainerStarted","Data":"44f84c34eaed19a05e06cd28b4c38fc4168f8fcdbaf4f4f97e2f3ee2fde5128a"} Nov 04 03:06:21.778654 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.778631 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" event={"ID":"0bd9f487-686c-4630-a511-03599a0e831e","Type":"ContainerStarted","Data":"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852"} Nov 04 03:06:21.778812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.778796 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" event={"ID":"0bd9f487-686c-4630-a511-03599a0e831e","Type":"ContainerStarted","Data":"ce1eb114c424a13971c8c48b08d53dece672dee3c3c3a6cc16634cd06646f1cf"} Nov 04 03:06:21.922628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:21.922560 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podStartSLOduration=1.922540138 podStartE2EDuration="1.922540138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:21.91380802 +0000 UTC m=+2028.880309013" watchObservedRunningTime="2026-11-04 03:06:21.922540138 +0000 UTC m=+2028.889041131" Nov 04 03:06:22.044702 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.044631 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podStartSLOduration=2.044608315 podStartE2EDuration="2.044608315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:22.037219867 +0000 UTC m=+2029.003720862" watchObservedRunningTime="2026-11-04 03:06:22.044608315 +0000 UTC m=+2029.011109308" Nov 04 03:06:22.095593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.095402 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:22.097649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.097616 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:22.097812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.097678 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:22.116927 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.116878 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:22.118482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.118451 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:22.118606 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:22.118511 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:23.106276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:23.105837 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:23.106276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:23.105903 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:23.116582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:23.116544 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:23.116755 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:23.116605 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:24.101116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:24.100654 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:24.101116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:24.100718 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:24.116227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:24.116148 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:24.116707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:24.116231 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:25.097051 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:25.096588 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:25.097051 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:25.096652 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:25.116101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:25.116058 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:25.116296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:25.116117 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:26.095737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:26.095696 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:26.096165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:26.096133 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:26.116732 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:26.116700 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:26.117255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:26.117226 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:27.131481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:27.126995 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:27.131481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:27.127057 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:27.146230 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:27.134546 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:27.146230 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:27.134609 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:28.105053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:28.105011 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 03:06:28.105415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:28.105367 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 03:06:28.116627 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:28.116597 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 04 03:06:28.116848 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:28.116826 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 04 03:06:29.124277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.120716 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:29.124277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.122508 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:29.135920 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.133612 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:06:29.141999 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.141298 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:29.144687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.143125 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:29.148539 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.148086 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:06:29.351555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.351511 4784 generic.go:356] "Generic (PLEG): container finished" podID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerID="a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0" exitCode=0 Nov 04 03:06:29.351769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.351577 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerDied","Data":"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0"} Nov 04 03:06:29.352146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.352123 4784 scope.go:122] "RemoveContainer" containerID="a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0" Nov 04 03:06:29.354238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.354155 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:29.354331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.354278 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:29.354331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:29.354320 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:06:30.388831 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:30.388797 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerStarted","Data":"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337"} Nov 04 03:06:31.301208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:31.301154 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:06:31.302703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:31.302675 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:06:31.302792 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:31.302742 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:06:31.986228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:31.976304 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:06:32.301263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:32.301136 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:06:32.301464 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:32.301282 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:06:33.301841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:33.301805 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:06:33.302278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:33.301890 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:06:34.300881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:34.300842 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:06:34.301092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:34.300911 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:06:35.300925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:35.300890 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 04 03:06:35.301506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:35.300950 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 04 03:06:35.955553 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:35.955511 4784 scope.go:122] "RemoveContainer" containerID="27c70b8a6600d5b24262b5c393f3345aa5d8a2225bbfcbc9daa1bd0213be66ae" Nov 04 03:06:36.306024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:36.305940 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:06:36.307118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:36.307096 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:06:36.402166 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:36.402131 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:06:39.062525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.062489 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x"] Nov 04 03:06:39.113573 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.113536 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d316993-6857x"] Nov 04 03:06:39.163291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.163259 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa54205-e485-4ad4-9cea-75a0b221aa19" path="/var/lib/kubelet/pods/3fa54205-e485-4ad4-9cea-75a0b221aa19/volumes" Nov 04 03:06:39.197951 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.197919 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 03:06:39.198501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.198465 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" containerID="cri-o://e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051" gracePeriod=30 Nov 04 03:06:39.205581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.205554 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 03:06:39.205828 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:39.205796 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" containerID="cri-o://19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18" gracePeriod=30 Nov 04 03:06:40.341678 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.341633 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:40.342212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.341715 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:40.663734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.663668 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 03:06:40.663962 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.663933 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" containerID="cri-o://27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377" gracePeriod=30 Nov 04 03:06:40.721380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.721343 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 03:06:40.721655 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.721606 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" containerID="cri-o://f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c" gracePeriod=30 Nov 04 03:06:40.886670 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.886613 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 03:06:40.886888 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.886850 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" containerID="cri-o://2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785" gracePeriod=30 Nov 04 03:06:40.891658 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.891628 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 03:06:40.891840 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:40.891819 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" containerID="cri-o://5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b" gracePeriod=30 Nov 04 03:06:41.928585 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:41.928542 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:41.928988 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:41.928603 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:42.093425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:42.093387 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:42.093609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:42.093454 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:42.747406 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:42.747369 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:42.747620 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:42.747443 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:43.182398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:43.182364 4784 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:43.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:43.182428 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:44.562634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.562606 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 03:06:44.589025 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.588998 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 03:06:44.652159 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652115 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652169 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652249 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652291 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652341 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652394 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652451 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.652491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652467 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652539 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652575 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652621 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652659 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652701 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652718 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652787 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652846 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652943 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\") pod \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\" (UID: \"3ef92cdc-24ad-48c1-a4eb-23fe719f4719\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652950 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.652988 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653001 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:44.653753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653020 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.653753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653032 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653108 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653171 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\") pod \"08292c14-0f3c-4af5-8972-94578b37990b\" (UID: \"08292c14-0f3c-4af5-8972-94578b37990b\") " Nov 04 03:06:44.653753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.653739 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:44.654221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654176 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654227 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654242 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654258 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654272 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654286 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654300 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08292c14-0f3c-4af5-8972-94578b37990b-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.654641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654593 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.655076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.654910 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.655617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.655262 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.655860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.655831 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.655946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.655833 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "kube-api-access-6xxb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.655946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.655836 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:44.657132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.657099 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.658302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.658276 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.658404 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.658277 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token" pod "3ef92cdc-24ad-48c1-a4eb-23fe719f4719" (UID: "3ef92cdc-24ad-48c1-a4eb-23fe719f4719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.659179 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.659114 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:44.659728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.659702 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p" pod "08292c14-0f3c-4af5-8972-94578b37990b" (UID: "08292c14-0f3c-4af5-8972-94578b37990b"). InnerVolumeSpecName "kube-api-access-w7d2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.729023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.728991 4784 generic.go:356] "Generic (PLEG): container finished" podID="08292c14-0f3c-4af5-8972-94578b37990b" containerID="19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18" exitCode=0 Nov 04 03:06:44.729267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.729042 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" Nov 04 03:06:44.729267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.729069 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" event={"ID":"08292c14-0f3c-4af5-8972-94578b37990b","Type":"ContainerDied","Data":"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18"} Nov 04 03:06:44.729267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.729130 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx" event={"ID":"08292c14-0f3c-4af5-8972-94578b37990b","Type":"ContainerDied","Data":"08de33a3d40da8cda142acb53b084a1dfa90c55fa2d636edbcba43647b131853"} Nov 04 03:06:44.729267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.729161 4784 scope.go:122] "RemoveContainer" containerID="19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18" Nov 04 03:06:44.731289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.731266 4784 generic.go:356] "Generic (PLEG): container finished" podID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerID="e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051" exitCode=0 Nov 04 03:06:44.731431 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.731306 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" Nov 04 03:06:44.731501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.731332 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" event={"ID":"3ef92cdc-24ad-48c1-a4eb-23fe719f4719","Type":"ContainerDied","Data":"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051"} Nov 04 03:06:44.731501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.731481 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl" event={"ID":"3ef92cdc-24ad-48c1-a4eb-23fe719f4719","Type":"ContainerDied","Data":"8588d90c0ec59a17218307e5d76dd5590756b921bde7916f71623d51a36099f4"} Nov 04 03:06:44.749261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.749238 4784 scope.go:122] "RemoveContainer" containerID="19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18" Nov 04 03:06:44.749716 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:44.749689 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18\": container with ID starting with 19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18 not found: ID does not exist" containerID="19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18" Nov 04 03:06:44.749807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.749729 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18"} err="failed to get container status \"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18\": rpc error: code = NotFound desc = could not find container \"19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18\": container with ID starting with 19b8acab3856701b43810af9e706264a82f1489c1f1c9f143b009de5b3f50f18 not found: ID does not exist" Nov 04 03:06:44.749807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.749755 4784 scope.go:122] "RemoveContainer" containerID="e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051" Nov 04 03:06:44.756444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756421 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756445 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756456 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxb2\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-kube-api-access-6xxb2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756466 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756475 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08292c14-0f3c-4af5-8972-94578b37990b-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756493 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756501 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef92cdc-24ad-48c1-a4eb-23fe719f4719-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756509 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7d2p\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-kube-api-access-w7d2p\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756518 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756525 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08292c14-0f3c-4af5-8972-94578b37990b-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.756613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.756534 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08292c14-0f3c-4af5-8972-94578b37990b-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.764755 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.764729 4784 scope.go:122] "RemoveContainer" containerID="e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051" Nov 04 03:06:44.765173 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:44.765140 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051\": container with ID starting with e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051 not found: ID does not exist" containerID="e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051" Nov 04 03:06:44.765321 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:44.765219 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051"} err="failed to get container status \"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051\": rpc error: code = NotFound desc = could not find container \"e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051\": container with ID starting with e22791f7b8e60dd4a4d4f6a58eeafc622dd691a7744d265feb3c592a806e4051 not found: ID does not exist" Nov 04 03:06:45.047295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:45.046776 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 03:06:45.063450 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:45.063408 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-fc8wx"] Nov 04 03:06:45.165023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:45.164984 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08292c14-0f3c-4af5-8972-94578b37990b" path="/var/lib/kubelet/pods/08292c14-0f3c-4af5-8972-94578b37990b/volumes" Nov 04 03:06:45.165603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:45.165575 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 03:06:45.165769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:45.165623 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-5mccl"] Nov 04 03:06:46.069218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.068534 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 03:06:46.082374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.082344 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 03:06:46.183369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.183336 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 03:06:46.189475 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189397 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.189630 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189484 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.189630 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189618 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.189757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189628 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.189757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189673 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.189757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189715 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.189929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189776 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.189929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189782 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.189929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189808 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.189929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189846 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.189929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189885 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.189944 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190010 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.190246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190110 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190148 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190257 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.190473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190329 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190367 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\") pod \"57bb2731-4c22-401e-9089-c9f8fe4d50de\" (UID: \"57bb2731-4c22-401e-9089-c9f8fe4d50de\") " Nov 04 03:06:46.190473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190426 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.190473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190461 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\") pod \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\" (UID: \"c15eff77-b23b-42aa-82b4-a69e34eee6cb\") " Nov 04 03:06:46.190735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190513 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.190735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190608 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.190915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.190892 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.191223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.191166 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.191594 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.191802 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.191951 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.191993 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192017 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192034 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192048 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bb2731-4c22-401e-9089-c9f8fe4d50de-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192062 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192074 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.192435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192173 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.193229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.192598 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.194770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.194635 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.194770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.194701 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.194770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.194713 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.195041 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.195016 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.195680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.195654 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.195843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.195820 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx" pod "57bb2731-4c22-401e-9089-c9f8fe4d50de" (UID: "57bb2731-4c22-401e-9089-c9f8fe4d50de"). InnerVolumeSpecName "kube-api-access-mxbmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.196383 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.196364 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "kube-api-access-8vvq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.197067 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.197045 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy" pod "c15eff77-b23b-42aa-82b4-a69e34eee6cb" (UID: "c15eff77-b23b-42aa-82b4-a69e34eee6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.292601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292573 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.292765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292630 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.292765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292675 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.292765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292737 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.292940 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292923 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.292985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.292968 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.293052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293029 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.293112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293042 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.293255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293119 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.293255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293158 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.293432 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293254 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\") pod \"95f70fa7-388f-45e7-bcb4-4687882be424\" (UID: \"95f70fa7-388f-45e7-bcb4-4687882be424\") " Nov 04 03:06:46.293432 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293308 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.293689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293465 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.293689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.293521 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.294024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294003 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294027 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvq7\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-kube-api-access-8vvq7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294041 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294052 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294062 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxbmx\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-kube-api-access-mxbmx\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294074 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294087 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294099 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294108 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294119 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294127 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bb2731-4c22-401e-9089-c9f8fe4d50de-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294135 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294146 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.294156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294160 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.295047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294168 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f70fa7-388f-45e7-bcb4-4687882be424-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.295047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.294176 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15eff77-b23b-42aa-82b4-a69e34eee6cb-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.295872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.295850 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.296007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.295977 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 03:06:46.296330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.296308 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.296706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.296682 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "kube-api-access-fds6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.296993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.296976 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo" pod "95f70fa7-388f-45e7-bcb4-4687882be424" (UID: "95f70fa7-388f-45e7-bcb4-4687882be424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.395203 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395152 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395271 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395295 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395445 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395482 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395545 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395581 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.395668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395602 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.395725 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395701 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.396276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.395911 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\") pod \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\" (UID: \"22e72fee-7ec0-4976-83d5-4b7438a0b8c7\") " Nov 04 03:06:46.396276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396084 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.396276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396158 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.396527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396319 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.396527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396404 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.396816 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396798 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396822 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396838 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fds6x\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-kube-api-access-fds6x\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396854 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396868 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f70fa7-388f-45e7-bcb4-4687882be424-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396883 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f70fa7-388f-45e7-bcb4-4687882be424-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396896 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396908 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f70fa7-388f-45e7-bcb4-4687882be424-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.396933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.396921 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.398359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.398336 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.398728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.398707 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.398835 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.398794 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.398835 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.398821 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc" pod "22e72fee-7ec0-4976-83d5-4b7438a0b8c7" (UID: "22e72fee-7ec0-4976-83d5-4b7438a0b8c7"). InnerVolumeSpecName "kube-api-access-ngssc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.499391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.499290 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.499391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.499336 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.499391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.499351 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngssc\" (UniqueName: \"kubernetes.io/projected/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-kube-api-access-ngssc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.499391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.499366 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e72fee-7ec0-4976-83d5-4b7438a0b8c7-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.799651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.799546 4784 generic.go:356] "Generic (PLEG): container finished" podID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerID="27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377" exitCode=0 Nov 04 03:06:46.799822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.799698 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" event={"ID":"57bb2731-4c22-401e-9089-c9f8fe4d50de","Type":"ContainerDied","Data":"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377"} Nov 04 03:06:46.799822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.799747 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" event={"ID":"57bb2731-4c22-401e-9089-c9f8fe4d50de","Type":"ContainerDied","Data":"a2e519dbb1ef3f129fd0f352d5d38cff04e9ce59e601dbe162df1cf891d18cfe"} Nov 04 03:06:46.799822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.799770 4784 scope.go:122] "RemoveContainer" containerID="27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377" Nov 04 03:06:46.800000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.799913 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm" Nov 04 03:06:46.806339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.805944 4784 generic.go:356] "Generic (PLEG): container finished" podID="95f70fa7-388f-45e7-bcb4-4687882be424" containerID="2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785" exitCode=0 Nov 04 03:06:46.806339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.806080 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" Nov 04 03:06:46.806339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.806125 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" event={"ID":"95f70fa7-388f-45e7-bcb4-4687882be424","Type":"ContainerDied","Data":"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785"} Nov 04 03:06:46.806339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.806168 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm" event={"ID":"95f70fa7-388f-45e7-bcb4-4687882be424","Type":"ContainerDied","Data":"2712dab8dc256238c228ac9cb83ead73cd8649629a4d73cdab2bfaaab2a6ce91"} Nov 04 03:06:46.811087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.811056 4784 generic.go:356] "Generic (PLEG): container finished" podID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerID="5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b" exitCode=0 Nov 04 03:06:46.811244 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.811223 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" event={"ID":"22e72fee-7ec0-4976-83d5-4b7438a0b8c7","Type":"ContainerDied","Data":"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b"} Nov 04 03:06:46.811345 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.811300 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" event={"ID":"22e72fee-7ec0-4976-83d5-4b7438a0b8c7","Type":"ContainerDied","Data":"c492a9b10ec987d8d7dc80e333c2e80daa1601859f93f7b0e9d8c492d4fdd6ee"} Nov 04 03:06:46.811414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.811405 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l" Nov 04 03:06:46.813419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.813249 4784 generic.go:356] "Generic (PLEG): container finished" podID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerID="f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c" exitCode=0 Nov 04 03:06:46.813419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.813291 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" Nov 04 03:06:46.813419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.813338 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" event={"ID":"c15eff77-b23b-42aa-82b4-a69e34eee6cb","Type":"ContainerDied","Data":"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c"} Nov 04 03:06:46.813419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.813393 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s" event={"ID":"c15eff77-b23b-42aa-82b4-a69e34eee6cb","Type":"ContainerDied","Data":"6b4e707b5162556c45bc79825d256b527a71f7c2fa784b9af2f3c6250742b9f6"} Nov 04 03:06:46.824053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.824028 4784 scope.go:122] "RemoveContainer" containerID="27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377" Nov 04 03:06:46.824696 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:46.824608 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377\": container with ID starting with 27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377 not found: ID does not exist" containerID="27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377" Nov 04 03:06:46.824696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.824649 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377"} err="failed to get container status \"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377\": rpc error: code = NotFound desc = could not find container \"27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377\": container with ID starting with 27b8bcdb6076c2e845c12ba2d95769dfd21cf46cd361c29e7b08be4f7ca74377 not found: ID does not exist" Nov 04 03:06:46.824696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.824668 4784 scope.go:122] "RemoveContainer" containerID="2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785" Nov 04 03:06:46.838893 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.838731 4784 scope.go:122] "RemoveContainer" containerID="2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785" Nov 04 03:06:46.839077 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:46.839054 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785\": container with ID starting with 2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785 not found: ID does not exist" containerID="2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785" Nov 04 03:06:46.839166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.839088 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785"} err="failed to get container status \"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785\": rpc error: code = NotFound desc = could not find container \"2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785\": container with ID starting with 2a6a92af420a4e2d68e33c7db0bfcb56f4e96101b213981081a05e87829b7785 not found: ID does not exist" Nov 04 03:06:46.839166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.839107 4784 scope.go:122] "RemoveContainer" containerID="5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b" Nov 04 03:06:46.850240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.850181 4784 scope.go:122] "RemoveContainer" containerID="5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b" Nov 04 03:06:46.850608 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:46.850585 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b\": container with ID starting with 5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b not found: ID does not exist" containerID="5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b" Nov 04 03:06:46.850729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.850622 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b"} err="failed to get container status \"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b\": rpc error: code = NotFound desc = could not find container \"5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b\": container with ID starting with 5cb7a8fe6cf547c3287fbc5bfdaeadaf6c8d6e26ffe3eaad0f5bb75be85cc29b not found: ID does not exist" Nov 04 03:06:46.850729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.850642 4784 scope.go:122] "RemoveContainer" containerID="f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c" Nov 04 03:06:46.869492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.869472 4784 scope.go:122] "RemoveContainer" containerID="f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c" Nov 04 03:06:46.869849 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:06:46.869827 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c\": container with ID starting with f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c not found: ID does not exist" containerID="f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c" Nov 04 03:06:46.869939 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.869858 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c"} err="failed to get container status \"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c\": rpc error: code = NotFound desc = could not find container \"f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c\": container with ID starting with f69fc245eafb709f7c875e2b610330f4c811ff828a867e69ead50655c17c290c not found: ID does not exist" Nov 04 03:06:46.976368 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.976128 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 03:06:46.984415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:46.984379 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-rsw2s"] Nov 04 03:06:47.090976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.090884 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 03:06:47.095661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.095634 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-52pkm"] Nov 04 03:06:47.164030 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.163998 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" path="/var/lib/kubelet/pods/3ef92cdc-24ad-48c1-a4eb-23fe719f4719/volumes" Nov 04 03:06:47.164560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.164545 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" path="/var/lib/kubelet/pods/95f70fa7-388f-45e7-bcb4-4687882be424/volumes" Nov 04 03:06:47.165034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.165019 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" path="/var/lib/kubelet/pods/c15eff77-b23b-42aa-82b4-a69e34eee6cb/volumes" Nov 04 03:06:47.209797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.209758 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 03:06:47.214336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.214308 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-czkcm"] Nov 04 03:06:47.345999 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.345914 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 03:06:47.385930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:47.385894 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-8kz5l"] Nov 04 03:06:49.163201 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:49.163142 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" path="/var/lib/kubelet/pods/22e72fee-7ec0-4976-83d5-4b7438a0b8c7/volumes" Nov 04 03:06:49.163899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:49.163875 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" path="/var/lib/kubelet/pods/57bb2731-4c22-401e-9089-c9f8fe4d50de/volumes" Nov 04 03:06:54.195507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:54.195462 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 03:06:54.196014 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:54.195734 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" containerID="cri-o://e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68" gracePeriod=30 Nov 04 03:06:56.819082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:56.819035 4784 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:56.819573 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:56.819102 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:59.490250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.490220 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 03:06:59.545985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.545951 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546089 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546156 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546209 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546249 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546318 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546379 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546413 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546447 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.546681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546545 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\") pod \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\" (UID: \"3f0589ee-8104-4e43-87f0-d32ab4d8bde1\") " Nov 04 03:06:59.546681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546575 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.546852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.546683 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:59.547146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547078 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.547146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547106 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.547453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547308 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.547453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547342 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.547603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547436 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.547603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547529 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.547603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.547544 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.548543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.548517 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.548716 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.548696 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:59.548790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.548776 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:59.549095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.549078 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj" pod "3f0589ee-8104-4e43-87f0-d32ab4d8bde1" (UID: "3f0589ee-8104-4e43-87f0-d32ab4d8bde1"). InnerVolumeSpecName "kube-api-access-xxjzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:59.649369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.649329 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjzj\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-kube-api-access-xxjzj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.649369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.649372 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.649605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.649388 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.649605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:06:59.649401 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0589ee-8104-4e43-87f0-d32ab4d8bde1-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:00.078471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.078437 4784 generic.go:356] "Generic (PLEG): container finished" podID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerID="e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68" exitCode=0 Nov 04 03:07:00.078675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.078488 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" Nov 04 03:07:00.078675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.078522 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" event={"ID":"3f0589ee-8104-4e43-87f0-d32ab4d8bde1","Type":"ContainerDied","Data":"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68"} Nov 04 03:07:00.078675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.078580 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5" event={"ID":"3f0589ee-8104-4e43-87f0-d32ab4d8bde1","Type":"ContainerDied","Data":"fefc50ead2f3f0e146ddf2da0dfc4f466c2e43b4fb75ffdc0a06339df293fd16"} Nov 04 03:07:00.078675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.078603 4784 scope.go:122] "RemoveContainer" containerID="e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68" Nov 04 03:07:00.091506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.091486 4784 scope.go:122] "RemoveContainer" containerID="e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68" Nov 04 03:07:00.091770 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:00.091750 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68\": container with ID starting with e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68 not found: ID does not exist" containerID="e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68" Nov 04 03:07:00.091818 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.091780 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68"} err="failed to get container status \"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68\": rpc error: code = NotFound desc = could not find container \"e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68\": container with ID starting with e2e20e64e49182c04c54a8215836fcf7488d7fe0e631f3ecbe29a29e0b99ab68 not found: ID does not exist" Nov 04 03:07:00.250208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.250156 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 03:07:00.301265 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:00.301219 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-8bhc5"] Nov 04 03:07:01.162459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:01.162423 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" path="/var/lib/kubelet/pods/3f0589ee-8104-4e43-87f0-d32ab4d8bde1/volumes" Nov 04 03:07:06.402393 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:06.402354 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:07:20.730678 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:20.730627 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:07:25.731333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:25.731279 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" podUID="0db5e7bc-970b-4751-b375-884f867fe031" containerName="payload-pre-processing" containerID="cri-o://6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6" gracePeriod=25 Nov 04 03:07:26.035244 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.035214 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:07:26.115734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.115697 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\" (UniqueName: \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\") pod \"0db5e7bc-970b-4751-b375-884f867fe031\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " Nov 04 03:07:26.115960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.115916 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\") pod \"0db5e7bc-970b-4751-b375-884f867fe031\" (UID: \"0db5e7bc-970b-4751-b375-884f867fe031\") " Nov 04 03:07:26.116306 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.116281 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume" pod "0db5e7bc-970b-4751-b375-884f867fe031" (UID: "0db5e7bc-970b-4751-b375-884f867fe031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:26.116663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.116639 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db5e7bc-970b-4751-b375-884f867fe031-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:26.118029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.118009 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv" pod "0db5e7bc-970b-4751-b375-884f867fe031" (UID: "0db5e7bc-970b-4751-b375-884f867fe031"). InnerVolumeSpecName "kube-api-access-6zsnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:26.218710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.218674 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zsnv\" (UniqueName: \"kubernetes.io/projected/0db5e7bc-970b-4751-b375-884f867fe031-kube-api-access-6zsnv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:26.487037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.487005 4784 generic.go:356] "Generic (PLEG): container finished" podID="0db5e7bc-970b-4751-b375-884f867fe031" containerID="6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6" exitCode=0 Nov 04 03:07:26.487253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.487077 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" Nov 04 03:07:26.487253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.487092 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" event={"ID":"0db5e7bc-970b-4751-b375-884f867fe031","Type":"ContainerDied","Data":"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6"} Nov 04 03:07:26.487253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.487166 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg" event={"ID":"0db5e7bc-970b-4751-b375-884f867fe031","Type":"ContainerDied","Data":"9d465f0ee46d5456dcad20004a4e766b6b3edbbd1ad2fb50c43b149e3f51779a"} Nov 04 03:07:26.487253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.487216 4784 scope.go:122] "RemoveContainer" containerID="6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6" Nov 04 03:07:26.502671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.502647 4784 scope.go:122] "RemoveContainer" containerID="6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6" Nov 04 03:07:26.503080 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:26.503052 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6\": container with ID starting with 6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6 not found: ID does not exist" containerID="6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6" Nov 04 03:07:26.503169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.503094 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6"} err="failed to get container status \"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6\": rpc error: code = NotFound desc = could not find container \"6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6\": container with ID starting with 6209542561e69bdba7afb40a2d9e83944040cebe255fb684212971c55ea3ccc6 not found: ID does not exist" Nov 04 03:07:26.661474 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.661438 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:07:26.714569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:26.714536 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-551e2b-869c56d547-549jg"] Nov 04 03:07:27.163992 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:27.163957 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db5e7bc-970b-4751-b375-884f867fe031" path="/var/lib/kubelet/pods/0db5e7bc-970b-4751-b375-884f867fe031/volumes" Nov 04 03:07:27.798614 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:27.798574 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:07:32.798981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:32.798914 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" podUID="e5b158c0-ea09-413b-a88a-3de8f11a3565" containerName="payload-processing" containerID="cri-o://d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2" gracePeriod=25 Nov 04 03:07:33.136929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.136881 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:07:33.292119 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.292079 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\") pod \"e5b158c0-ea09-413b-a88a-3de8f11a3565\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " Nov 04 03:07:33.292345 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.292319 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\" (UniqueName: \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\") pod \"e5b158c0-ea09-413b-a88a-3de8f11a3565\" (UID: \"e5b158c0-ea09-413b-a88a-3de8f11a3565\") " Nov 04 03:07:33.292560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.292535 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume" pod "e5b158c0-ea09-413b-a88a-3de8f11a3565" (UID: "e5b158c0-ea09-413b-a88a-3de8f11a3565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:33.293306 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.293282 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b158c0-ea09-413b-a88a-3de8f11a3565-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:33.296264 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.296238 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr" pod "e5b158c0-ea09-413b-a88a-3de8f11a3565" (UID: "e5b158c0-ea09-413b-a88a-3de8f11a3565"). InnerVolumeSpecName "kube-api-access-gdbmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:33.394906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.394809 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdbmr\" (UniqueName: \"kubernetes.io/projected/e5b158c0-ea09-413b-a88a-3de8f11a3565-kube-api-access-gdbmr\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:33.582420 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.582387 4784 generic.go:356] "Generic (PLEG): container finished" podID="e5b158c0-ea09-413b-a88a-3de8f11a3565" containerID="d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2" exitCode=0 Nov 04 03:07:33.582617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.582434 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" Nov 04 03:07:33.582617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.582474 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" event={"ID":"e5b158c0-ea09-413b-a88a-3de8f11a3565","Type":"ContainerDied","Data":"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2"} Nov 04 03:07:33.582617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.582534 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7" event={"ID":"e5b158c0-ea09-413b-a88a-3de8f11a3565","Type":"ContainerDied","Data":"94b2fb92d7c9d93cdd61054ced04456dbc1420fde176b893383c62181c9f9573"} Nov 04 03:07:33.582617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.582560 4784 scope.go:122] "RemoveContainer" containerID="d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2" Nov 04 03:07:33.594833 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.594807 4784 scope.go:122] "RemoveContainer" containerID="d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2" Nov 04 03:07:33.595232 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:33.595173 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2\": container with ID starting with d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2 not found: ID does not exist" containerID="d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2" Nov 04 03:07:33.595364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.595245 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2"} err="failed to get container status \"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2\": rpc error: code = NotFound desc = could not find container \"d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2\": container with ID starting with d79be80c559c6ce3355ed0350495ef75c80c4769c0b4bac439d301d950946fe2 not found: ID does not exist" Nov 04 03:07:33.729922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.729895 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:07:33.733970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.733944 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:07:33.735372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.735334 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:07:33.739720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.739697 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:07:33.753601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.753565 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:07:33.804929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:33.804896 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675c80-698585c7d-b5hn7"] Nov 04 03:07:35.179793 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:35.179753 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b158c0-ea09-413b-a88a-3de8f11a3565" path="/var/lib/kubelet/pods/e5b158c0-ea09-413b-a88a-3de8f11a3565/volumes" Nov 04 03:07:36.000993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:36.000959 4784 scope.go:122] "RemoveContainer" containerID="6cfb367f752c3ba16d9c0793209b50dec560f90b28087e034f277ee64a1d9904" Nov 04 03:07:36.403159 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:36.403076 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:07:37.226456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:37.226415 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:07:37.226676 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:37.226502 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:07:39.643146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.643107 4784 generic.go:356] "Generic (PLEG): container finished" podID="b12d7995-7b2a-406c-9c56-e0a899176738" containerID="0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa" exitCode=0 Nov 04 03:07:39.643881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.643254 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerDied","Data":"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa"} Nov 04 03:07:39.643942 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.643826 4784 scope.go:122] "RemoveContainer" containerID="0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa" Nov 04 03:07:39.646486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.646445 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:07:39.646640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.646615 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:07:39.646698 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:39.646676 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:07:40.653944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:40.653891 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerStarted","Data":"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619"} Nov 04 03:07:41.435822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:41.435779 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:41.436024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:41.435995 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 04 03:07:41.436080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:41.436061 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 04 03:07:42.436705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:42.436610 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 04 03:07:42.436705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:42.436673 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 04 03:07:43.436269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:43.436233 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 04 03:07:43.436461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:43.436296 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 04 03:07:44.435982 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:44.435947 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 04 03:07:44.436460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:44.436009 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 04 03:07:45.436143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:45.436107 4784 patch_prober.go:29] interesting pod/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 04 03:07:45.436603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:45.436176 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 04 03:07:46.440842 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:46.440810 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:46.441333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:46.441311 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:46.442007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:46.441987 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:50.235415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:50.235378 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:07:50.235944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:50.235627 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" containerID="cri-o://3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619" gracePeriod=30 Nov 04 03:07:55.535808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.535773 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:55.613272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613238 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613463 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613338 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613463 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613418 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613524 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613554 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613639 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613683 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613724 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.613781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613752 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:07:55.614537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613764 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\") pod \"b12d7995-7b2a-406c-9c56-e0a899176738\" (UID: \"b12d7995-7b2a-406c-9c56-e0a899176738\") " Nov 04 03:07:55.614537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613843 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:55.614537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.613998 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:07:55.614537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614264 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:07:55.614537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614389 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:07:55.614955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614928 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.615028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614960 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.615028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614977 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.615028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.614992 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d7995-7b2a-406c-9c56-e0a899176738-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.615028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.615006 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.616004 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.615980 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:55.616115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.616081 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "kube-api-access-fvq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:55.616328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.616312 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:07:55.616534 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.616517 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo" pod "b12d7995-7b2a-406c-9c56-e0a899176738" (UID: "b12d7995-7b2a-406c-9c56-e0a899176738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:07:55.716724 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.716688 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d7995-7b2a-406c-9c56-e0a899176738-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.716724 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.716724 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.716724 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.716739 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d7995-7b2a-406c-9c56-e0a899176738-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.717038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.716755 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq4t\" (UniqueName: \"kubernetes.io/projected/b12d7995-7b2a-406c-9c56-e0a899176738-kube-api-access-fvq4t\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:55.810804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.810693 4784 generic.go:356] "Generic (PLEG): container finished" podID="b12d7995-7b2a-406c-9c56-e0a899176738" containerID="3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619" exitCode=0 Nov 04 03:07:55.810804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.810745 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" Nov 04 03:07:55.810804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.810782 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerDied","Data":"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619"} Nov 04 03:07:55.811162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.810850 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt" event={"ID":"b12d7995-7b2a-406c-9c56-e0a899176738","Type":"ContainerDied","Data":"03da50c09055dcd241fcc1ea6369922f3660ee8d4338eb0bb9bb3d8383f5a220"} Nov 04 03:07:55.811162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.810878 4784 scope.go:122] "RemoveContainer" containerID="3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619" Nov 04 03:07:55.824780 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.824751 4784 scope.go:122] "RemoveContainer" containerID="0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa" Nov 04 03:07:55.838169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.838145 4784 scope.go:122] "RemoveContainer" containerID="3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619" Nov 04 03:07:55.838599 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:55.838578 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619\": container with ID starting with 3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619 not found: ID does not exist" containerID="3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619" Nov 04 03:07:55.838685 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.838615 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619"} err="failed to get container status \"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619\": rpc error: code = NotFound desc = could not find container \"3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619\": container with ID starting with 3de1cdb8e6b301ec619ed774e1fd9b1cd7d04e758305a54bafe1e1d38f1f8619 not found: ID does not exist" Nov 04 03:07:55.838685 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.838644 4784 scope.go:122] "RemoveContainer" containerID="0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa" Nov 04 03:07:55.838933 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:07:55.838912 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa\": container with ID starting with 0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa not found: ID does not exist" containerID="0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa" Nov 04 03:07:55.838980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.838942 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa"} err="failed to get container status \"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa\": rpc error: code = NotFound desc = could not find container \"0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa\": container with ID starting with 0c8ae183733d6b35f7b1deb0c413e9b391d04afa9f842f802c4e1302fc3bd7fa not found: ID does not exist" Nov 04 03:07:55.982990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.982948 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:07:55.990212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:55.990166 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-675c80-openshift-default-554bc6cd5c-fsdlt"] Nov 04 03:07:57.162601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:07:57.162561 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" path="/var/lib/kubelet/pods/b12d7995-7b2a-406c-9c56-e0a899176738/volumes" Nov 04 03:08:00.637950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:00.637909 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 03:08:00.748779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:00.748745 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 03:08:05.638965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.638879 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" podUID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" containerName="payload-processing" containerID="cri-o://ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e" gracePeriod=25 Nov 04 03:08:05.749170 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.749103 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" podUID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" containerName="payload-pre-processing" containerID="cri-o://d3a20d7e835e75ae4f0a0129390a9eb81ffdec3240a6338a8e31f4f4057015e3" gracePeriod=25 Nov 04 03:08:05.941650 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.941616 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 03:08:05.978833 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.977987 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\" (UniqueName: \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\") pod \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " Nov 04 03:08:05.978833 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.978115 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\") pod \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\" (UID: \"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c\") " Nov 04 03:08:05.979338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.979302 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume" pod "3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" (UID: "3c4774a3-13c9-4ca3-bd3f-59c816db2f8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:05.981557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:05.981515 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7" pod "3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" (UID: "3c4774a3-13c9-4ca3-bd3f-59c816db2f8c"). InnerVolumeSpecName "kube-api-access-2snp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:06.054537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.054500 4784 generic.go:356] "Generic (PLEG): container finished" podID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" containerID="ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e" exitCode=0 Nov 04 03:08:06.054744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.054543 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" event={"ID":"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c","Type":"ContainerDied","Data":"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e"} Nov 04 03:08:06.054744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.054568 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" Nov 04 03:08:06.054744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.054616 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf" event={"ID":"3c4774a3-13c9-4ca3-bd3f-59c816db2f8c","Type":"ContainerDied","Data":"37387ab291c1f80d28314d5a7f6c72034b0a2ca5ac1f3fccae056f4f5f8cd757"} Nov 04 03:08:06.054744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.054641 4784 scope.go:122] "RemoveContainer" containerID="ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e" Nov 04 03:08:06.056560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.056499 4784 generic.go:356] "Generic (PLEG): container finished" podID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" containerID="d3a20d7e835e75ae4f0a0129390a9eb81ffdec3240a6338a8e31f4f4057015e3" exitCode=0 Nov 04 03:08:06.056686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.056531 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" event={"ID":"6c68f76e-7ba8-4aca-b778-e7b113cc460f","Type":"ContainerDied","Data":"d3a20d7e835e75ae4f0a0129390a9eb81ffdec3240a6338a8e31f4f4057015e3"} Nov 04 03:08:06.058309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.058290 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 03:08:06.080611 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.080568 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2snp7\" (UniqueName: \"kubernetes.io/projected/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-kube-api-access-2snp7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:06.080752 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.080620 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:06.080812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.080761 4784 scope.go:122] "RemoveContainer" containerID="ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e" Nov 04 03:08:06.081258 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:06.081182 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e\": container with ID starting with ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e not found: ID does not exist" containerID="ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e" Nov 04 03:08:06.081397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.081268 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e"} err="failed to get container status \"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e\": rpc error: code = NotFound desc = could not find container \"ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e\": container with ID starting with ebf8b65081a70ff18cbf2ae034a32f34e20a0a51db237ef54a07176b4299728e not found: ID does not exist" Nov 04 03:08:06.181301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.181263 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\") pod \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " Nov 04 03:08:06.181498 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.181396 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\") pod \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\" (UID: \"6c68f76e-7ba8-4aca-b778-e7b113cc460f\") " Nov 04 03:08:06.181953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.181926 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume" pod "6c68f76e-7ba8-4aca-b778-e7b113cc460f" (UID: "6c68f76e-7ba8-4aca-b778-e7b113cc460f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:06.184683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.184652 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh" pod "6c68f76e-7ba8-4aca-b778-e7b113cc460f" (UID: "6c68f76e-7ba8-4aca-b778-e7b113cc460f"). InnerVolumeSpecName "kube-api-access-qhjvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:06.283970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.283929 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/6c68f76e-7ba8-4aca-b778-e7b113cc460f-kube-api-access-qhjvh\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:06.283970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.283970 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c68f76e-7ba8-4aca-b778-e7b113cc460f-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:06.354740 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.354696 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 03:08:06.361253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:06.361220 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-297806-6b77744954-k4jwf"] Nov 04 03:08:06.405352 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:06.405306 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:08:07.065929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.065896 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" Nov 04 03:08:07.065929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.065909 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw" event={"ID":"6c68f76e-7ba8-4aca-b778-e7b113cc460f","Type":"ContainerDied","Data":"f596d30fd14c81a31e0ea24abe7a5423b0c1631676208b89eefeff4e57acc131"} Nov 04 03:08:07.066557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.065981 4784 scope.go:122] "RemoveContainer" containerID="d3a20d7e835e75ae4f0a0129390a9eb81ffdec3240a6338a8e31f4f4057015e3" Nov 04 03:08:07.162669 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.162635 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" path="/var/lib/kubelet/pods/3c4774a3-13c9-4ca3-bd3f-59c816db2f8c/volumes" Nov 04 03:08:07.244234 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.244176 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 03:08:07.294825 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:07.294783 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-297806-bdfd869bc-8j6sw"] Nov 04 03:08:09.087864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.087772 4784 generic.go:356] "Generic (PLEG): container finished" podID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerID="ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c" exitCode=0 Nov 04 03:08:09.087864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.087845 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerDied","Data":"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c"} Nov 04 03:08:09.088383 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.088366 4784 scope.go:122] "RemoveContainer" containerID="ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c" Nov 04 03:08:09.090141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.090112 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:09.090223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.090197 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:09.090266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.090234 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:09.162620 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:09.162588 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" path="/var/lib/kubelet/pods/6c68f76e-7ba8-4aca-b778-e7b113cc460f/volumes" Nov 04 03:08:10.098563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:10.098531 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerStarted","Data":"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa"} Nov 04 03:08:10.777222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:10.773652 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:08:10.777222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:10.773785 4784 patch_prober.go:29] interesting pod/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 03:08:10.777222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:10.773841 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 03:08:11.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:11.773686 4784 patch_prober.go:29] interesting pod/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 03:08:11.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:11.773761 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 03:08:12.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:12.772673 4784 patch_prober.go:29] interesting pod/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 03:08:12.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:12.772737 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 03:08:13.775215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:13.775022 4784 patch_prober.go:29] interesting pod/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 03:08:13.775215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:13.775086 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 03:08:14.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:14.772898 4784 patch_prober.go:29] interesting pod/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 03:08:14.774215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:14.772969 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 03:08:15.777306 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:15.777276 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:08:15.777703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:15.777587 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:08:15.778252 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:15.778236 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:08:20.050593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:20.050556 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h"] Nov 04 03:08:20.069846 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:20.069800 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95359c5c-dr78h"] Nov 04 03:08:21.162911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:21.162873 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba05c3c-d631-4117-a835-ca11ea6a9acd" path="/var/lib/kubelet/pods/aba05c3c-d631-4117-a835-ca11ea6a9acd/volumes" Nov 04 03:08:26.932679 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:26.932643 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:08:27.047943 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:27.047903 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:08:30.837148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:30.837112 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:08:30.837636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:30.837340 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" containerID="cri-o://85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b" gracePeriod=30 Nov 04 03:08:31.933872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:31.933803 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" podUID="1a7d0f9a-db83-4269-996e-20754563d353" containerName="payload-processing" containerID="cri-o://e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132" gracePeriod=25 Nov 04 03:08:32.048624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.048569 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" podUID="a786813b-3dfa-405d-be13-45061bf3e640" containerName="payload-pre-processing" containerID="cri-o://4b304a731c69ecb37ebc21c3eb6d5ecbfc8bc02089ef2b5b37a5d0b37c302a4e" gracePeriod=25 Nov 04 03:08:32.229581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.229548 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:08:32.296092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.296053 4784 generic.go:356] "Generic (PLEG): container finished" podID="a786813b-3dfa-405d-be13-45061bf3e640" containerID="4b304a731c69ecb37ebc21c3eb6d5ecbfc8bc02089ef2b5b37a5d0b37c302a4e" exitCode=0 Nov 04 03:08:32.296384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.296309 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" event={"ID":"a786813b-3dfa-405d-be13-45061bf3e640","Type":"ContainerDied","Data":"4b304a731c69ecb37ebc21c3eb6d5ecbfc8bc02089ef2b5b37a5d0b37c302a4e"} Nov 04 03:08:32.298341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.298315 4784 generic.go:356] "Generic (PLEG): container finished" podID="1a7d0f9a-db83-4269-996e-20754563d353" containerID="e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132" exitCode=0 Nov 04 03:08:32.298513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.298367 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" Nov 04 03:08:32.298513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.298383 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" event={"ID":"1a7d0f9a-db83-4269-996e-20754563d353","Type":"ContainerDied","Data":"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132"} Nov 04 03:08:32.298513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.298428 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt" event={"ID":"1a7d0f9a-db83-4269-996e-20754563d353","Type":"ContainerDied","Data":"85fef03a554a59132d997cc52a1d7c923fafd1992842703569f5dea56a1d1e4a"} Nov 04 03:08:32.298513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.298458 4784 scope.go:122] "RemoveContainer" containerID="e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132" Nov 04 03:08:32.323992 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.323957 4784 scope.go:122] "RemoveContainer" containerID="e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132" Nov 04 03:08:32.332247 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:32.325166 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132\": container with ID starting with e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132 not found: ID does not exist" containerID="e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132" Nov 04 03:08:32.332247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.325256 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132"} err="failed to get container status \"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132\": rpc error: code = NotFound desc = could not find container \"e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132\": container with ID starting with e8062ca673ae373918a8dddcbb8c2d67f3866e2dd7f9aec714b134a2ef70e132 not found: ID does not exist" Nov 04 03:08:32.357324 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.357250 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:08:32.394989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.394953 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume" pod "1a7d0f9a-db83-4269-996e-20754563d353" (UID: "1a7d0f9a-db83-4269-996e-20754563d353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:32.395256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.394402 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\") pod \"1a7d0f9a-db83-4269-996e-20754563d353\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " Nov 04 03:08:32.396043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.396001 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\") pod \"1a7d0f9a-db83-4269-996e-20754563d353\" (UID: \"1a7d0f9a-db83-4269-996e-20754563d353\") " Nov 04 03:08:32.397112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.397088 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d0f9a-db83-4269-996e-20754563d353-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:32.399551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.399525 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb" pod "1a7d0f9a-db83-4269-996e-20754563d353" (UID: "1a7d0f9a-db83-4269-996e-20754563d353"). InnerVolumeSpecName "kube-api-access-v7mnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:32.516259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.511944 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\" (UniqueName: \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\") pod \"a786813b-3dfa-405d-be13-45061bf3e640\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " Nov 04 03:08:32.516259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.512247 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\") pod \"a786813b-3dfa-405d-be13-45061bf3e640\" (UID: \"a786813b-3dfa-405d-be13-45061bf3e640\") " Nov 04 03:08:32.516259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.513124 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mnb\" (UniqueName: \"kubernetes.io/projected/1a7d0f9a-db83-4269-996e-20754563d353-kube-api-access-v7mnb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:32.542227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.540731 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume" pod "a786813b-3dfa-405d-be13-45061bf3e640" (UID: "a786813b-3dfa-405d-be13-45061bf3e640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:32.562786 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.562741 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc" pod "a786813b-3dfa-405d-be13-45061bf3e640" (UID: "a786813b-3dfa-405d-be13-45061bf3e640"). InnerVolumeSpecName "kube-api-access-8jsqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:32.617975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.616337 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsqc\" (UniqueName: \"kubernetes.io/projected/a786813b-3dfa-405d-be13-45061bf3e640-kube-api-access-8jsqc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:32.617975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.616373 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786813b-3dfa-405d-be13-45061bf3e640-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:32.758275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.758241 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:08:32.762546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:32.762513 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-569fb7-69bcd6f796-49qwt"] Nov 04 03:08:33.164695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.164665 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7d0f9a-db83-4269-996e-20754563d353" path="/var/lib/kubelet/pods/1a7d0f9a-db83-4269-996e-20754563d353/volumes" Nov 04 03:08:33.222746 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.222664 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx"] Nov 04 03:08:33.223628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223606 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.223696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223632 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.223696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223666 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" Nov 04 03:08:33.223696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223673 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" Nov 04 03:08:33.223696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223683 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" containerName="payload-processing" Nov 04 03:08:33.223696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223692 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" containerName="payload-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223706 4784 cpu_manager.go:414] "containerMap: removing container" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223712 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223725 4784 cpu_manager.go:414] "containerMap: removing container" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223732 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223740 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223746 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223759 4784 cpu_manager.go:414] "containerMap: removing container" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223766 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223787 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223794 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223807 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0db5e7bc-970b-4751-b375-884f867fe031" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223815 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db5e7bc-970b-4751-b375-884f867fe031" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223833 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a786813b-3dfa-405d-be13-45061bf3e640" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223841 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a786813b-3dfa-405d-be13-45061bf3e640" containerName="payload-pre-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223852 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1a7d0f9a-db83-4269-996e-20754563d353" containerName="payload-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223859 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7d0f9a-db83-4269-996e-20754563d353" containerName="payload-processing" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223867 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223874 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.223899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223898 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e5b158c0-ea09-413b-a88a-3de8f11a3565" containerName="payload-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223910 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b158c0-ea09-413b-a88a-3de8f11a3565" containerName="payload-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223927 4784 cpu_manager.go:414] "containerMap: removing container" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223933 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223947 4784 cpu_manager.go:414] "containerMap: removing container" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.223953 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224111 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a786813b-3dfa-405d-be13-45061bf3e640" containerName="payload-pre-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224124 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7d0f9a-db83-4269-996e-20754563d353" containerName="payload-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224131 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224143 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b158c0-ea09-413b-a88a-3de8f11a3565" containerName="payload-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224152 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0589ee-8104-4e43-87f0-d32ab4d8bde1" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224160 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bb2731-4c22-401e-9089-c9f8fe4d50de" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224172 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12d7995-7b2a-406c-9c56-e0a899176738" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224183 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15eff77-b23b-42aa-82b4-a69e34eee6cb" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224208 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4774a3-13c9-4ca3-bd3f-59c816db2f8c" containerName="payload-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224217 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f70fa7-388f-45e7-bcb4-4687882be424" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224225 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e72fee-7ec0-4976-83d5-4b7438a0b8c7" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224234 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef92cdc-24ad-48c1-a4eb-23fe719f4719" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224242 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="08292c14-0f3c-4af5-8972-94578b37990b" containerName="istio-proxy" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224253 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c68f76e-7ba8-4aca-b778-e7b113cc460f" containerName="payload-pre-processing" Nov 04 03:08:33.224560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.224262 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db5e7bc-970b-4751-b375-884f867fe031" containerName="payload-pre-processing" Nov 04 03:08:33.228223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.228204 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.282059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.282006 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:08:33.282059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.282038 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:08:33.286478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.286399 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx"] Nov 04 03:08:33.312743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.312713 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:08:33.312959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.312927 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:08:33.319235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.319162 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" Nov 04 03:08:33.319235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.319175 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz" event={"ID":"a786813b-3dfa-405d-be13-45061bf3e640","Type":"ContainerDied","Data":"412b8e5097a26ca0a45b754b278595d8fb1dc74fde09e70c844155afd3bc1fc8"} Nov 04 03:08:33.319525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.319265 4784 scope.go:122] "RemoveContainer" containerID="4b304a731c69ecb37ebc21c3eb6d5ecbfc8bc02089ef2b5b37a5d0b37c302a4e" Nov 04 03:08:33.326173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.326145 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.326369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.326236 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbgf\" (UniqueName: \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.326369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.326346 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.427874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.427837 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.428109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.428092 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.428259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.428244 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbgf\" (UniqueName: \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.428745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.428721 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.430823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.430801 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.461765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.461721 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:08:33.466037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.466007 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-569fb7-8594f58547-sfqmz"] Nov 04 03:08:33.491213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.491116 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbgf\" (UniqueName: \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.543508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.543470 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:33.886969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.886883 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:08:33.892480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.892459 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941162 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941337 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941382 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941421 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941487 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941573 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941691 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5lf\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941753 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:33.944236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:33.941781 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.033211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.027683 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-db5ef5-openshift-default-dockercfg-tc7b7\"" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052310 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052377 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052417 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052529 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052643 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5lf\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052707 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052737 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052812 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.052960 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.054916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.053521 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.143993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.105558 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.143993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.106294 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.143993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.106624 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.143993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.106861 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.143993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.107079 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.144619 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.144345 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx"] Nov 04 03:08:34.152613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.152531 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:08:34.220299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.220234 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.224258 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.222436 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.224258 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.223944 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5lf\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\") pod \"e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:34.337247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.333721 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" event={"ID":"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2","Type":"ContainerStarted","Data":"90a6932d712a4706b0ecd85c3466517b17e96c659d61d97a757ed79ad6a6bec2"} Nov 04 03:08:34.519473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:34.519430 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:35.066785 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.066746 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:08:35.076955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.076906 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:35.077100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.077012 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:35.077100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.077063 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:35.215235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.214981 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a786813b-3dfa-405d-be13-45061bf3e640" path="/var/lib/kubelet/pods/a786813b-3dfa-405d-be13-45061bf3e640/volumes" Nov 04 03:08:35.394754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.394652 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerStarted","Data":"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4"} Nov 04 03:08:35.395264 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.395068 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerStarted","Data":"71f78f8239234f929627d3aac05739a4aad3e39dad01c9eaa7d364f56c0a55f3"} Nov 04 03:08:35.397659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.397394 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" event={"ID":"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2","Type":"ContainerStarted","Data":"82eb1b3c6d139e7efa713ea34f72b1cf6277e043ed018bc6ad3315b288257140"} Nov 04 03:08:35.521527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.520111 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:35.521527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.520629 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podStartSLOduration=2.5205928010000003 podStartE2EDuration="2.520592801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:08:35.520467736 +0000 UTC m=+2162.486968730" watchObservedRunningTime="2026-11-04 03:08:35.520592801 +0000 UTC m=+2162.487093795" Nov 04 03:08:35.535144 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.535108 4784 patch_prober.go:29] interesting pod/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:08:35.535404 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.535165 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:08:35.656269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:35.652402 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" podStartSLOduration=2.652381972 podStartE2EDuration="2.652381972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:08:35.648333455 +0000 UTC m=+2162.614834451" watchObservedRunningTime="2026-11-04 03:08:35.652381972 +0000 UTC m=+2162.618882966" Nov 04 03:08:35.906214 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:08:35.897755 4784 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35dceb6_82b8_49ab_8983_d4fbf58d8fe9.slice/crio-f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35dceb6_82b8_49ab_8983_d4fbf58d8fe9.slice/crio-f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35dceb6_82b8_49ab_8983_d4fbf58d8fe9.slice/crio-f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990/memory.stat: no such device], continuing to push stats Nov 04 03:08:36.246260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.246153 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:08:36.260098 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260061 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260127 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260181 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260241 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260278 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260295 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260375 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260502 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260542 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\") pod \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\" (UID: \"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9\") " Nov 04 03:08:36.260730 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260697 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:36.260791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260746 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:36.260791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260755 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:36.261028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.260984 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:36.261336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.261268 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.261336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.261303 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.261336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.261317 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.261336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.261331 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.261734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.261402 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:36.262981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.262953 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:08:36.263121 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.263102 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:36.263347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.263323 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:36.263428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.263377 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj" pod "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" (UID: "e35dceb6-82b8-49ab-8983-d4fbf58d8fe9"). InnerVolumeSpecName "kube-api-access-6ffrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:36.363379 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.363339 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.363379 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.363380 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.363607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.363397 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffrj\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-kube-api-access-6ffrj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.363607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.363414 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.363607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.363429 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:36.408268 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:36.408233 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:08:36.409374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.409346 4784 generic.go:356] "Generic (PLEG): container finished" podID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerID="85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b" exitCode=0 Nov 04 03:08:36.409517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.409427 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" Nov 04 03:08:36.409517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.409469 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerDied","Data":"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b"} Nov 04 03:08:36.409628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.409543 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m" event={"ID":"e35dceb6-82b8-49ab-8983-d4fbf58d8fe9","Type":"ContainerDied","Data":"f9e701741148855fee1d385acb61fe82ead473afd18c20562ac692c88c179990"} Nov 04 03:08:36.409628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.409581 4784 scope.go:122] "RemoveContainer" containerID="85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b" Nov 04 03:08:36.413157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.412178 4784 generic.go:356] "Generic (PLEG): container finished" podID="cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" containerID="82eb1b3c6d139e7efa713ea34f72b1cf6277e043ed018bc6ad3315b288257140" exitCode=0 Nov 04 03:08:36.413157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.412270 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" event={"ID":"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2","Type":"ContainerDied","Data":"82eb1b3c6d139e7efa713ea34f72b1cf6277e043ed018bc6ad3315b288257140"} Nov 04 03:08:36.425013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.424991 4784 scope.go:122] "RemoveContainer" containerID="5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c" Nov 04 03:08:36.436361 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.436339 4784 scope.go:122] "RemoveContainer" containerID="85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b" Nov 04 03:08:36.436661 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:36.436634 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b\": container with ID starting with 85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b not found: ID does not exist" containerID="85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b" Nov 04 03:08:36.436707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.436674 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b"} err="failed to get container status \"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b\": rpc error: code = NotFound desc = could not find container \"85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b\": container with ID starting with 85b69c967f2532edec26b0c26d282dc600337b126ca477f361cfa7ff78c36b0b not found: ID does not exist" Nov 04 03:08:36.436707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.436693 4784 scope.go:122] "RemoveContainer" containerID="5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c" Nov 04 03:08:36.436942 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:36.436928 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c\": container with ID starting with 5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c not found: ID does not exist" containerID="5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c" Nov 04 03:08:36.436990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.436945 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c"} err="failed to get container status \"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c\": rpc error: code = NotFound desc = could not find container \"5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c\": container with ID starting with 5981830195a9e10620270bebcdeeba49ad74e1bba6be5cacb1314cfdaa0d8b5c not found: ID does not exist" Nov 04 03:08:36.526748 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.526673 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:36.582484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.582452 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:08:36.587659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:36.587628 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cdb11d-openshift-default-5ff9547799-2x65m"] Nov 04 03:08:37.164459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.164424 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" path="/var/lib/kubelet/pods/e35dceb6-82b8-49ab-8983-d4fbf58d8fe9/volumes" Nov 04 03:08:37.423461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.423112 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:37.424357 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.424338 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:08:37.612741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.612712 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:37.687526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.687424 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\") pod \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " Nov 04 03:08:37.687708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.687606 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\" (UniqueName: \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\") pod \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " Nov 04 03:08:37.687708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.687662 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\") pod \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\" (UID: \"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2\") " Nov 04 03:08:37.688124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.688093 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca" pod "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" (UID: "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:37.688449 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.688322 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:37.689971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.689941 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca" pod "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" (UID: "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:08:37.690115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.690092 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf" pod "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" (UID: "cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2"). InnerVolumeSpecName "kube-api-access-jvbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:37.789773 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.789725 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:37.789773 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:37.789767 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbgf\" (UniqueName: \"kubernetes.io/projected/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2-kube-api-access-jvbgf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:38.433720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:38.433681 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" Nov 04 03:08:38.434136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:38.433717 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx" event={"ID":"cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2","Type":"ContainerDied","Data":"90a6932d712a4706b0ecd85c3466517b17e96c659d61d97a757ed79ad6a6bec2"} Nov 04 03:08:38.434136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:38.433766 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a6932d712a4706b0ecd85c3466517b17e96c659d61d97a757ed79ad6a6bec2" Nov 04 03:08:40.824967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:40.824912 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:08:40.825536 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:40.825278 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" containerID="cri-o://50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a" gracePeriod=30 Nov 04 03:08:43.232712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:43.232680 4784 patch_prober.go:29] interesting pod/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:08:43.233101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:43.232740 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:08:46.247289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.247254 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:08:46.319257 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319223 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319466 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319290 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319466 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319367 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319472 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319571 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319599 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319598 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:46.319699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319650 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319657 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:46.319917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319732 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319766 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\") pod \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\" (UID: \"928e8f79-c330-4fea-8eaf-2a55de6d7cbb\") " Nov 04 03:08:46.319917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319858 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:46.320084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.319951 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:46.320505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320479 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:46.320716 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320696 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.320771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320731 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.320771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320747 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.320771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320762 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.320924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.320777 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.321953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.321924 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:08:46.322052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.321985 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:08:46.322272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.322258 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:46.322385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.322371 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422" pod "928e8f79-c330-4fea-8eaf-2a55de6d7cbb" (UID: "928e8f79-c330-4fea-8eaf-2a55de6d7cbb"). InnerVolumeSpecName "kube-api-access-5b422". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:46.422115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.422031 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.422115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.422063 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.422115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.422075 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.422115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.422090 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b422\" (UniqueName: \"kubernetes.io/projected/928e8f79-c330-4fea-8eaf-2a55de6d7cbb-kube-api-access-5b422\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:46.513667 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.513629 4784 generic.go:356] "Generic (PLEG): container finished" podID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerID="50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a" exitCode=0 Nov 04 03:08:46.513851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.513692 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" Nov 04 03:08:46.513851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.513712 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerDied","Data":"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a"} Nov 04 03:08:46.513851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.513775 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt" event={"ID":"928e8f79-c330-4fea-8eaf-2a55de6d7cbb","Type":"ContainerDied","Data":"b0c10234a1bd4cb88fa9197cfe0de9790c0b20c0ac8dd97dc61fda88169ddd9a"} Nov 04 03:08:46.513851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.513801 4784 scope.go:122] "RemoveContainer" containerID="50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a" Nov 04 03:08:46.525531 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.525509 4784 scope.go:122] "RemoveContainer" containerID="be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd" Nov 04 03:08:46.539442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.539416 4784 scope.go:122] "RemoveContainer" containerID="50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a" Nov 04 03:08:46.539892 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:46.539869 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a\": container with ID starting with 50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a not found: ID does not exist" containerID="50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a" Nov 04 03:08:46.539985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.539905 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a"} err="failed to get container status \"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a\": rpc error: code = NotFound desc = could not find container \"50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a\": container with ID starting with 50cb602090971cf43e2f3ad385cea337693e707d295bfee49375de93c8666f7a not found: ID does not exist" Nov 04 03:08:46.539985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.539934 4784 scope.go:122] "RemoveContainer" containerID="be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd" Nov 04 03:08:46.540219 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:46.540202 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd\": container with ID starting with be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd not found: ID does not exist" containerID="be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd" Nov 04 03:08:46.540265 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.540225 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd"} err="failed to get container status \"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd\": rpc error: code = NotFound desc = could not find container \"be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd\": container with ID starting with be23e2e420e997d315c41401acdb136b6645308b3894f663dd4431d6c2c421fd not found: ID does not exist" Nov 04 03:08:46.686871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.686791 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:08:46.691054 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:46.691028 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-551e2b-openshift-default-5cb6c84ff7-bxgnt"] Nov 04 03:08:47.191006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:47.190970 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" path="/var/lib/kubelet/pods/928e8f79-c330-4fea-8eaf-2a55de6d7cbb/volumes" Nov 04 03:08:48.642211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.641818 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642884 4784 cpu_manager.go:414] "containerMap: removing container" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642908 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642930 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642939 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642952 4784 cpu_manager.go:414] "containerMap: removing container" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642960 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642977 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.642984 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643036 4784 cpu_manager.go:414] "containerMap: removing container" podUID="cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" containerName="revoke-keys" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643044 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" containerName="revoke-keys" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643274 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" containerName="revoke-keys" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643322 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643333 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643344 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="928e8f79-c330-4fea-8eaf-2a55de6d7cbb" containerName="istio-proxy" Nov 04 03:08:48.648549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.643356 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35dceb6-82b8-49ab-8983-d4fbf58d8fe9" containerName="istio-proxy" Nov 04 03:08:48.651124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.651088 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:48.857509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.700859 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:08:48.857509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.729418 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-db5ef5-dockercfg-p485p\"" Nov 04 03:08:48.857509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.729530 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-db5ef5\"" Nov 04 03:08:48.857509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.829036 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:08:48.926765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.926718 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:48.988582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:48.987176 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwmn\" (UniqueName: \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.009599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.007609 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:08:49.009599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.007731 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.088787 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.088635 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwmn\" (UniqueName: \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.088787 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.088743 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8h7c\" (UniqueName: \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.088787 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.088764 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.089303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.088814 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.089529 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.089510 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.150916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.150884 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwmn\" (UniqueName: \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\") pod \"payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.189950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.189848 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8h7c\" (UniqueName: \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.189950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.189895 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.191918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.191884 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.253009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.252970 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8h7c\" (UniqueName: \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\") pod \"payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.310543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.310503 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:49.329482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.329453 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:49.738373 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.738297 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:08:49.796780 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:49.796750 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:08:49.798512 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:08:49.798279 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2eadc5_f33e_4517_831d_adfd898c068e.slice/crio-f436ddbd33f09e8c8135e19f1ec06cf8155e177f30ab76c31e6490f55e7f15ee WatchSource:0}: Error finding container f436ddbd33f09e8c8135e19f1ec06cf8155e177f30ab76c31e6490f55e7f15ee: Status 404 returned error can't find the container with id f436ddbd33f09e8c8135e19f1ec06cf8155e177f30ab76c31e6490f55e7f15ee Nov 04 03:08:50.681141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.681097 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" event={"ID":"9d2eadc5-f33e-4517-831d-adfd898c068e","Type":"ContainerStarted","Data":"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07"} Nov 04 03:08:50.681894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.681863 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" event={"ID":"9d2eadc5-f33e-4517-831d-adfd898c068e","Type":"ContainerStarted","Data":"f436ddbd33f09e8c8135e19f1ec06cf8155e177f30ab76c31e6490f55e7f15ee"} Nov 04 03:08:50.682507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.682479 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:08:50.683782 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.683761 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" event={"ID":"635ee2a1-40eb-4e16-9738-3b7eaa7d321f","Type":"ContainerStarted","Data":"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be"} Nov 04 03:08:50.683880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.683798 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" event={"ID":"635ee2a1-40eb-4e16-9738-3b7eaa7d321f","Type":"ContainerStarted","Data":"2413e2e3c4cd0439f76b265fb5a57b331c7f813734bac81478a8c25c0c0027d6"} Nov 04 03:08:50.683880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.683832 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:08:50.804090 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:50.804036 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" podStartSLOduration=2.804022186 podStartE2EDuration="2.804022186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:08:50.800145777 +0000 UTC m=+2177.766646795" watchObservedRunningTime="2026-11-04 03:08:50.804022186 +0000 UTC m=+2177.770523178" Nov 04 03:08:53.196560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:53.196475 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" podStartSLOduration=5.196452887 podStartE2EDuration="5.196452887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:08:50.921334448 +0000 UTC m=+2177.887835445" watchObservedRunningTime="2026-11-04 03:08:53.196452887 +0000 UTC m=+2180.162953883" Nov 04 03:08:53.199508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:53.199478 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:08:53.265710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:53.265672 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:08:57.753613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.753579 4784 generic.go:356] "Generic (PLEG): container finished" podID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerID="ccc80c34705355278a817cc9f4c77051f872faf20d0c6592b0cf621165ac004a" exitCode=0 Nov 04 03:08:57.754502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.753665 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerDied","Data":"ccc80c34705355278a817cc9f4c77051f872faf20d0c6592b0cf621165ac004a"} Nov 04 03:08:57.754502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.754247 4784 scope.go:122] "RemoveContainer" containerID="ccc80c34705355278a817cc9f4c77051f872faf20d0c6592b0cf621165ac004a" Nov 04 03:08:57.756082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.756052 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:57.756233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.756125 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:57.756233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:57.756160 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:08:58.199871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.199812 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" podUID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" containerName="payload-processing" containerID="cri-o://536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e" gracePeriod=25 Nov 04 03:08:58.266739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.266665 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" podUID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" containerName="payload-pre-processing" containerID="cri-o://b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd" gracePeriod=25 Nov 04 03:08:58.609007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.606378 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:08:58.703470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.703436 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:08:58.789316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.789219 4784 generic.go:356] "Generic (PLEG): container finished" podID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" containerID="536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e" exitCode=0 Nov 04 03:08:58.789316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.789275 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" Nov 04 03:08:58.789866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.789303 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" event={"ID":"ef06c0b7-2af6-4949-9777-0c71c78e65c2","Type":"ContainerDied","Data":"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e"} Nov 04 03:08:58.789866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.789377 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g" event={"ID":"ef06c0b7-2af6-4949-9777-0c71c78e65c2","Type":"ContainerDied","Data":"ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112"} Nov 04 03:08:58.789866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.789413 4784 scope.go:122] "RemoveContainer" containerID="536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e" Nov 04 03:08:58.791487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.791463 4784 generic.go:356] "Generic (PLEG): container finished" podID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" containerID="b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd" exitCode=0 Nov 04 03:08:58.791617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.791548 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" Nov 04 03:08:58.791681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.791546 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" event={"ID":"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e","Type":"ContainerDied","Data":"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd"} Nov 04 03:08:58.791749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.791710 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq" event={"ID":"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e","Type":"ContainerDied","Data":"a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba"} Nov 04 03:08:58.793386 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.793361 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\") pod \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " Nov 04 03:08:58.793545 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.793524 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\" (UniqueName: \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\") pod \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\" (UID: \"ef06c0b7-2af6-4949-9777-0c71c78e65c2\") " Nov 04 03:08:58.795270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.795133 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume" pod "ef06c0b7-2af6-4949-9777-0c71c78e65c2" (UID: "ef06c0b7-2af6-4949-9777-0c71c78e65c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:58.795971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.795938 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerStarted","Data":"655124c487d1ced04bb29744d3b26da2abefc858ddd83fd6aaf5a6a446076cf1"} Nov 04 03:08:58.796792 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.796767 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5" pod "ef06c0b7-2af6-4949-9777-0c71c78e65c2" (UID: "ef06c0b7-2af6-4949-9777-0c71c78e65c2"). InnerVolumeSpecName "kube-api-access-h5wx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:58.808720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.808699 4784 scope.go:122] "RemoveContainer" containerID="536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e" Nov 04 03:08:58.809081 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:58.809059 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e\": container with ID starting with 536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e not found: ID does not exist" containerID="536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e" Nov 04 03:08:58.809173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.809099 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e"} err="failed to get container status \"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e\": rpc error: code = NotFound desc = could not find container \"536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e\": container with ID starting with 536962ae9f48e12188b29b06498242288498b8d59f56a46f72349db07bce142e not found: ID does not exist" Nov 04 03:08:58.809173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.809124 4784 scope.go:122] "RemoveContainer" containerID="b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd" Nov 04 03:08:58.820661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.820635 4784 scope.go:122] "RemoveContainer" containerID="b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd" Nov 04 03:08:58.820988 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:58.820960 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd\": container with ID starting with b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd not found: ID does not exist" containerID="b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd" Nov 04 03:08:58.821068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.821002 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd"} err="failed to get container status \"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd\": rpc error: code = NotFound desc = could not find container \"b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd\": container with ID starting with b040e13fda41b53c2e72ad9091cd817029105f3b7fb0f2634b026b4a3f4838fd not found: ID does not exist" Nov 04 03:08:58.895022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.894973 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\") pod \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " Nov 04 03:08:58.895382 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.895358 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\" (UniqueName: \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\") pod \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\" (UID: \"1dc78c6d-584a-4bb6-aceb-e986fbae8e2e\") " Nov 04 03:08:58.895514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.895485 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume" pod "1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" (UID: "1dc78c6d-584a-4bb6-aceb-e986fbae8e2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:58.896333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.896254 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef06c0b7-2af6-4949-9777-0c71c78e65c2-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:58.896333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.896284 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:58.896333 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.896300 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5wx5\" (UniqueName: \"kubernetes.io/projected/ef06c0b7-2af6-4949-9777-0c71c78e65c2-kube-api-access-h5wx5\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:58.898481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.898451 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4" pod "1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" (UID: "1dc78c6d-584a-4bb6-aceb-e986fbae8e2e"). InnerVolumeSpecName "kube-api-access-jmdc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:58.998091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:58.998055 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdc4\" (UniqueName: \"kubernetes.io/projected/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e-kube-api-access-jmdc4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:59.142954 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:08:59.142914 4784 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc78c6d_584a_4bb6_aceb_e986fbae8e2e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc78c6d_584a_4bb6_aceb_e986fbae8e2e.slice/crio-a1475fbfa832ef4388e4f5cb149efdbe4a7d72bc3eab21490b0ea32d6a0a4cba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef06c0b7_2af6_4949_9777_0c71c78e65c2.slice/crio-ff1e149df03f8345366b1e10071a0e2593034c960f8b3c40ffc0637bb33e7112\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef06c0b7_2af6_4949_9777_0c71c78e65c2.slice\": RecentStats: unable to find data in memory cache]" Nov 04 03:08:59.149555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.149525 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:08:59.149739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.149719 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:08:59.149796 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.149765 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:08:59.242557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.242524 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:08:59.246737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.246694 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-4b2bf4-6cd5fc4b6c-wwqfq"] Nov 04 03:08:59.384514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.384398 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:08:59.389408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:08:59.389374 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-4b2bf4-69448f59dd-fqq6g"] Nov 04 03:09:00.150269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:00.150232 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:09:00.150715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:00.150292 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:09:01.150358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:01.150321 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:09:01.150784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:01.150379 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:09:01.192353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:01.184169 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" path="/var/lib/kubelet/pods/1dc78c6d-584a-4bb6-aceb-e986fbae8e2e/volumes" Nov 04 03:09:01.192353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:01.185040 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" path="/var/lib/kubelet/pods/ef06c0b7-2af6-4949-9777-0c71c78e65c2/volumes" Nov 04 03:09:01.697253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:01.696825 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:09:02.150273 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:02.149808 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:09:02.150273 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:02.149867 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:09:02.708228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:02.706405 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:09:03.095925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.095829 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:09:03.096135 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.096106 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" containerID="cri-o://d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c" gracePeriod=30 Nov 04 03:09:03.101428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.101398 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:09:03.101610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.101589 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" containerID="cri-o://3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852" gracePeriod=30 Nov 04 03:09:03.149646 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.149611 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:09:03.149833 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:03.149668 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:09:04.150610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:04.150569 4784 patch_prober.go:29] interesting pod/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 03:09:04.151069 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:04.150632 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 03:09:05.163670 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.163640 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:09:05.164263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.164179 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:09:05.165413 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.165390 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:09:05.736384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.736345 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv"] Nov 04 03:09:05.737126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737109 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" containerName="payload-pre-processing" Nov 04 03:09:05.737126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737127 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" containerName="payload-pre-processing" Nov 04 03:09:05.737302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737144 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" containerName="payload-processing" Nov 04 03:09:05.737302 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737150 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" containerName="payload-processing" Nov 04 03:09:05.737370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737318 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc78c6d-584a-4bb6-aceb-e986fbae8e2e" containerName="payload-pre-processing" Nov 04 03:09:05.737370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.737329 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef06c0b7-2af6-4949-9777-0c71c78e65c2" containerName="payload-processing" Nov 04 03:09:05.742531 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.742511 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.782239 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.782179 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxjf\" (UniqueName: \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.782434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.782344 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.782594 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.782572 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.796652 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.796617 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:09:05.796879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.796624 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:09:05.796879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.796633 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:09:05.797113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.796672 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:09:05.797880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.797855 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv"] Nov 04 03:09:05.884050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.884007 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.884309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.884128 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxjf\" (UniqueName: \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.884309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.884274 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.884891 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.884866 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.886696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.886676 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:05.947064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:05.947025 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxjf\" (UniqueName: \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\") pod \"maas-api-revoke-sub-44249e5b-prgbv\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:06.058145 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:06.058045 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:06.402035 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:06.401939 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:09:06.444831 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:09:06.444800 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e8a204_95ba_41a4_8a4f_142432a04b91.slice/crio-d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec WatchSource:0}: Error finding container d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec: Status 404 returned error can't find the container with id d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec Nov 04 03:09:06.445794 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:06.445765 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv"] Nov 04 03:09:06.864675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:06.864637 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" event={"ID":"f5e8a204-95ba-41a4-8a4f-142432a04b91","Type":"ContainerStarted","Data":"7b9aa6cfd50fc0b2afd6cf17458013de9e339a91ae3a340d22d4817a7164a5b3"} Nov 04 03:09:06.864871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:06.864721 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" event={"ID":"f5e8a204-95ba-41a4-8a4f-142432a04b91","Type":"ContainerStarted","Data":"d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec"} Nov 04 03:09:06.985761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:06.985655 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" podStartSLOduration=1.985639649 podStartE2EDuration="1.985639649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:06.983103255 +0000 UTC m=+2193.949604239" watchObservedRunningTime="2026-11-04 03:09:06.985639649 +0000 UTC m=+2193.952140641" Nov 04 03:09:07.873952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:07.873854 4784 generic.go:356] "Generic (PLEG): container finished" podID="f5e8a204-95ba-41a4-8a4f-142432a04b91" containerID="7b9aa6cfd50fc0b2afd6cf17458013de9e339a91ae3a340d22d4817a7164a5b3" exitCode=0 Nov 04 03:09:07.873952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:07.873938 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" event={"ID":"f5e8a204-95ba-41a4-8a4f-142432a04b91","Type":"ContainerDied","Data":"7b9aa6cfd50fc0b2afd6cf17458013de9e339a91ae3a340d22d4817a7164a5b3"} Nov 04 03:09:08.396281 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.396175 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:09:08.440507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.440481 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:09:08.527115 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527077 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.527342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527217 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.527342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527253 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.527342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527282 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.527342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527308 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.527571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527401 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.527571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527429 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.527571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527444 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.527571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527476 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.527571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527561 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527564 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527632 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527690 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527734 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527799 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527852 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527862 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527927 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527941 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.527965 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528040 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528259 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528262 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528457 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528479 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528512 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\") pod \"0bd9f487-686c-4630-a511-03599a0e831e\" (UID: \"0bd9f487-686c-4630-a511-03599a0e831e\") " Nov 04 03:09:08.528616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.528550 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\") pod \"45024334-79f6-4211-bdf2-9c5f4106dd59\" (UID: \"45024334-79f6-4211-bdf2-9c5f4106dd59\") " Nov 04 03:09:08.529704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529654 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529679 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd9f487-686c-4630-a511-03599a0e831e-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529694 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529710 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529726 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529741 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529755 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45024334-79f6-4211-bdf2-9c5f4106dd59-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529769 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.529879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.529784 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.530328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.530296 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.531246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.530962 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.531246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.530971 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.531246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.530904 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "kube-api-access-lp468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.531246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.530990 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "kube-api-access-tjgl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.531246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.531173 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.532502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.532482 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.532903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.532873 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo" pod "0bd9f487-686c-4630-a511-03599a0e831e" (UID: "0bd9f487-686c-4630-a511-03599a0e831e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:08.532985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.532932 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo" pod "45024334-79f6-4211-bdf2-9c5f4106dd59" (UID: "45024334-79f6-4211-bdf2-9c5f4106dd59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:08.630859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630822 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp468\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-kube-api-access-lp468\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.630859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630855 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd9f487-686c-4630-a511-03599a0e831e-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.630859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630865 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630875 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630887 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630896 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgl5\" (UniqueName: \"kubernetes.io/projected/0bd9f487-686c-4630-a511-03599a0e831e-kube-api-access-tjgl5\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630906 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630914 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd9f487-686c-4630-a511-03599a0e831e-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.631109 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.630922 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45024334-79f6-4211-bdf2-9c5f4106dd59-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.883296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.883264 4784 generic.go:356] "Generic (PLEG): container finished" podID="0bd9f487-686c-4630-a511-03599a0e831e" containerID="3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852" exitCode=0 Nov 04 03:09:08.883791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.883313 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" Nov 04 03:09:08.883791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.883358 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" event={"ID":"0bd9f487-686c-4630-a511-03599a0e831e","Type":"ContainerDied","Data":"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852"} Nov 04 03:09:08.883791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.883435 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv" event={"ID":"0bd9f487-686c-4630-a511-03599a0e831e","Type":"ContainerDied","Data":"ce1eb114c424a13971c8c48b08d53dece672dee3c3c3a6cc16634cd06646f1cf"} Nov 04 03:09:08.883791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.883469 4784 scope.go:122] "RemoveContainer" containerID="3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852" Nov 04 03:09:08.885125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.885098 4784 generic.go:356] "Generic (PLEG): container finished" podID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerID="d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c" exitCode=0 Nov 04 03:09:08.885267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.885151 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" Nov 04 03:09:08.885267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.885175 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" event={"ID":"45024334-79f6-4211-bdf2-9c5f4106dd59","Type":"ContainerDied","Data":"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c"} Nov 04 03:09:08.885267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.885239 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc" event={"ID":"45024334-79f6-4211-bdf2-9c5f4106dd59","Type":"ContainerDied","Data":"44f84c34eaed19a05e06cd28b4c38fc4168f8fcdbaf4f4f97e2f3ee2fde5128a"} Nov 04 03:09:08.897710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.897684 4784 scope.go:122] "RemoveContainer" containerID="3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852" Nov 04 03:09:08.898059 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:08.898038 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852\": container with ID starting with 3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852 not found: ID does not exist" containerID="3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852" Nov 04 03:09:08.898143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.898072 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852"} err="failed to get container status \"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852\": rpc error: code = NotFound desc = could not find container \"3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852\": container with ID starting with 3e719666cd6261e5e5cbe0c8498e12673964f8a21f985cb8c2479ee25fe07852 not found: ID does not exist" Nov 04 03:09:08.898143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.898097 4784 scope.go:122] "RemoveContainer" containerID="d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c" Nov 04 03:09:08.912236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.912212 4784 scope.go:122] "RemoveContainer" containerID="d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c" Nov 04 03:09:08.912613 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:08.912590 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c\": container with ID starting with d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c not found: ID does not exist" containerID="d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c" Nov 04 03:09:08.912694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:08.912625 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c"} err="failed to get container status \"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c\": rpc error: code = NotFound desc = could not find container \"d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c\": container with ID starting with d595ae4c31d6d790911c245328cf309880f2013b2b6b9c99373d4039ab03320c not found: ID does not exist" Nov 04 03:09:09.040343 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.040313 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:09.057127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.057094 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:09:09.107678 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.107642 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-5zskv"] Nov 04 03:09:09.138013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.137934 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\") pod \"f5e8a204-95ba-41a4-8a4f-142432a04b91\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " Nov 04 03:09:09.138013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.137983 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\") pod \"f5e8a204-95ba-41a4-8a4f-142432a04b91\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " Nov 04 03:09:09.138213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.138144 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\" (UniqueName: \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\") pod \"f5e8a204-95ba-41a4-8a4f-142432a04b91\" (UID: \"f5e8a204-95ba-41a4-8a4f-142432a04b91\") " Nov 04 03:09:09.138456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.138428 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca" pod "f5e8a204-95ba-41a4-8a4f-142432a04b91" (UID: "f5e8a204-95ba-41a4-8a4f-142432a04b91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:09.138803 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.138782 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.140470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.140448 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca" pod "f5e8a204-95ba-41a4-8a4f-142432a04b91" (UID: "f5e8a204-95ba-41a4-8a4f-142432a04b91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:09.140601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.140584 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf" pod "f5e8a204-95ba-41a4-8a4f-142432a04b91" (UID: "f5e8a204-95ba-41a4-8a4f-142432a04b91"). InnerVolumeSpecName "kube-api-access-ckxjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:09.163236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.163175 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd9f487-686c-4630-a511-03599a0e831e" path="/var/lib/kubelet/pods/0bd9f487-686c-4630-a511-03599a0e831e/volumes" Nov 04 03:09:09.174896 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.174863 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:09:09.179027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.179003 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-vfrcc"] Nov 04 03:09:09.240384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.240328 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxjf\" (UniqueName: \"kubernetes.io/projected/f5e8a204-95ba-41a4-8a4f-142432a04b91-kube-api-access-ckxjf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.240384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.240363 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e8a204-95ba-41a4-8a4f-142432a04b91-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.895149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.895114 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" Nov 04 03:09:09.895739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.895163 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv" event={"ID":"f5e8a204-95ba-41a4-8a4f-142432a04b91","Type":"ContainerDied","Data":"d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec"} Nov 04 03:09:09.895739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:09.895273 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d976c6ba5d0a98126838711a60f975e159eb9ed94e150a05cfe8ce58b55066ec" Nov 04 03:09:10.018634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:10.018595 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 03:09:10.044611 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:10.044572 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 03:09:11.162368 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:11.162333 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" path="/var/lib/kubelet/pods/45024334-79f6-4211-bdf2-9c5f4106dd59/volumes" Nov 04 03:09:17.556235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556116 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl"] Nov 04 03:09:17.556811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556797 4784 cpu_manager.go:414] "containerMap: removing container" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" Nov 04 03:09:17.556864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556812 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" Nov 04 03:09:17.556864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556828 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" Nov 04 03:09:17.556864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556833 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" Nov 04 03:09:17.556960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556880 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f5e8a204-95ba-41a4-8a4f-142432a04b91" containerName="revoke-keys" Nov 04 03:09:17.556960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556886 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e8a204-95ba-41a4-8a4f-142432a04b91" containerName="revoke-keys" Nov 04 03:09:17.557024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556985 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e8a204-95ba-41a4-8a4f-142432a04b91" containerName="revoke-keys" Nov 04 03:09:17.557024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.556993 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="45024334-79f6-4211-bdf2-9c5f4106dd59" containerName="istio-proxy" Nov 04 03:09:17.557024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.557001 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd9f487-686c-4630-a511-03599a0e831e" containerName="istio-proxy" Nov 04 03:09:17.562083 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.562064 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.618957 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.618922 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl"] Nov 04 03:09:17.663879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.663755 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:09:17.663879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.663767 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:09:17.663879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.663793 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:09:17.663879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.663846 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:09:17.664535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.664511 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.664624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.664558 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj645\" (UniqueName: \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.664624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.664580 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.765864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.765826 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.766101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.765882 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj645\" (UniqueName: \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.766101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.765902 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.766679 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.766655 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.768437 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.768417 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.828749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.828673 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj645\" (UniqueName: \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\") pod \"maas-api-revoke-sub-adb708d2-p9hxl\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:17.877870 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:17.877825 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:18.266550 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:09:18.266511 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4799f0b_61e8_4b28_b9b5_29f80077a213.slice/crio-f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560 WatchSource:0}: Error finding container f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560: Status 404 returned error can't find the container with id f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560 Nov 04 03:09:18.268475 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:18.268442 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl"] Nov 04 03:09:18.982736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:18.982660 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" event={"ID":"a4799f0b-61e8-4b28-b9b5-29f80077a213","Type":"ContainerStarted","Data":"9fd08b09b0ebbc2e1e78801cd491ac64e85fb14bc33c40ace2213d47be088699"} Nov 04 03:09:18.983295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:18.982806 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" event={"ID":"a4799f0b-61e8-4b28-b9b5-29f80077a213","Type":"ContainerStarted","Data":"f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560"} Nov 04 03:09:19.016672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:19.016638 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 03:09:19.132809 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:19.132752 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" podStartSLOduration=2.132734434 podStartE2EDuration="2.132734434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:19.129171019 +0000 UTC m=+2206.095672011" watchObservedRunningTime="2026-11-04 03:09:19.132734434 +0000 UTC m=+2206.099235426" Nov 04 03:09:19.994593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:19.994557 4784 generic.go:356] "Generic (PLEG): container finished" podID="a4799f0b-61e8-4b28-b9b5-29f80077a213" containerID="9fd08b09b0ebbc2e1e78801cd491ac64e85fb14bc33c40ace2213d47be088699" exitCode=0 Nov 04 03:09:19.995043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:19.994641 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" event={"ID":"a4799f0b-61e8-4b28-b9b5-29f80077a213","Type":"ContainerDied","Data":"9fd08b09b0ebbc2e1e78801cd491ac64e85fb14bc33c40ace2213d47be088699"} Nov 04 03:09:21.176992 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.176959 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:21.237746 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.237709 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\") pod \"a4799f0b-61e8-4b28-b9b5-29f80077a213\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " Nov 04 03:09:21.237942 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.237790 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\" (UniqueName: \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\") pod \"a4799f0b-61e8-4b28-b9b5-29f80077a213\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " Nov 04 03:09:21.237942 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.237867 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\") pod \"a4799f0b-61e8-4b28-b9b5-29f80077a213\" (UID: \"a4799f0b-61e8-4b28-b9b5-29f80077a213\") " Nov 04 03:09:21.238155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.238131 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca" pod "a4799f0b-61e8-4b28-b9b5-29f80077a213" (UID: "a4799f0b-61e8-4b28-b9b5-29f80077a213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:21.238496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.238452 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:21.240111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.240082 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca" pod "a4799f0b-61e8-4b28-b9b5-29f80077a213" (UID: "a4799f0b-61e8-4b28-b9b5-29f80077a213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:21.240279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.240181 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645" pod "a4799f0b-61e8-4b28-b9b5-29f80077a213" (UID: "a4799f0b-61e8-4b28-b9b5-29f80077a213"). InnerVolumeSpecName "kube-api-access-kj645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:21.340378 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.340279 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj645\" (UniqueName: \"kubernetes.io/projected/a4799f0b-61e8-4b28-b9b5-29f80077a213-kube-api-access-kj645\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:21.340378 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:21.340316 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4799f0b-61e8-4b28-b9b5-29f80077a213-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:22.016869 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:22.016837 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" Nov 04 03:09:22.017111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:22.017021 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl" event={"ID":"a4799f0b-61e8-4b28-b9b5-29f80077a213","Type":"ContainerDied","Data":"f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560"} Nov 04 03:09:22.017111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:22.017078 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f749fe2ccc700151838ef771e2429dda6c352fdac7d6512a3b374bc9aa23c560" Nov 04 03:09:24.017905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:24.017863 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" containerID="cri-o://1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec" gracePeriod=25 Nov 04 03:09:25.043533 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.043491 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" containerID="cri-o://cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3" gracePeriod=45 Nov 04 03:09:25.070082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.070032 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" containerID="cri-o://d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca" gracePeriod=45 Nov 04 03:09:25.360947 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.360576 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 03:09:25.390165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.390135 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 03:09:25.500413 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500379 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.500662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500478 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\" (UniqueName: \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.500662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500571 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.500662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500604 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\" (UniqueName: \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.500849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500710 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.500849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500770 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.500849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500805 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.501009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500865 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.501009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500922 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.501009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500951 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.501009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.500986 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.501326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.501030 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\") pod \"15a23946-92d8-4c17-ac49-799f536621a5\" (UID: \"15a23946-92d8-4c17-ac49-799f536621a5\") " Nov 04 03:09:25.501326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.501084 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\") pod \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\" (UID: \"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02\") " Nov 04 03:09:25.502547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.502515 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.503766 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.503522 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.504200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.503927 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.504200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.504007 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.504392 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.504338 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.504741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.504710 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "kube-api-access-4zlwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:25.504901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.504880 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:25.508116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.508079 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.508524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.508495 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.508642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.508572 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm" pod "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" (UID: "8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:25.508761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.508741 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "kube-api-access-99lgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:25.509828 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.509802 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs" pod "15a23946-92d8-4c17-ac49-799f536621a5" (UID: "15a23946-92d8-4c17-ac49-799f536621a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615313 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zlwc\" (UniqueName: \"kubernetes.io/projected/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-kube-api-access-4zlwc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615349 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615365 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99lgb\" (UniqueName: \"kubernetes.io/projected/15a23946-92d8-4c17-ac49-799f536621a5-kube-api-access-99lgb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615381 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615406 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.615442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615438 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.618808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615452 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.618808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615466 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a23946-92d8-4c17-ac49-799f536621a5-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.618808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615479 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.618808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615492 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a23946-92d8-4c17-ac49-799f536621a5-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:25.618808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:25.615507 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:26.050297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.050261 4784 generic.go:356] "Generic (PLEG): container finished" podID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerID="d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca" exitCode=0 Nov 04 03:09:26.050809 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.050310 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" Nov 04 03:09:26.050809 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.050336 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" event={"ID":"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02","Type":"ContainerDied","Data":"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca"} Nov 04 03:09:26.050809 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.050388 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt" event={"ID":"8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02","Type":"ContainerDied","Data":"f11f95288272d392e0e28564d8462a3c792df987df861c652a7b963bd54a3247"} Nov 04 03:09:26.050809 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.050406 4784 scope.go:122] "RemoveContainer" containerID="d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca" Nov 04 03:09:26.052374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.052346 4784 generic.go:356] "Generic (PLEG): container finished" podID="15a23946-92d8-4c17-ac49-799f536621a5" containerID="cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3" exitCode=0 Nov 04 03:09:26.052482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.052398 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" Nov 04 03:09:26.052482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.052436 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" event={"ID":"15a23946-92d8-4c17-ac49-799f536621a5","Type":"ContainerDied","Data":"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3"} Nov 04 03:09:26.052589 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.052478 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm" event={"ID":"15a23946-92d8-4c17-ac49-799f536621a5","Type":"ContainerDied","Data":"9f702616270d28062db8f0f2439fa8410412125dba8642f53e996245bce049d1"} Nov 04 03:09:26.076022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.076003 4784 scope.go:122] "RemoveContainer" containerID="d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca" Nov 04 03:09:26.076371 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:26.076350 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca\": container with ID starting with d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca not found: ID does not exist" containerID="d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca" Nov 04 03:09:26.076428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.076380 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca"} err="failed to get container status \"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca\": rpc error: code = NotFound desc = could not find container \"d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca\": container with ID starting with d129a3623e30439dbb15792ba6af42a62799b99b517c9cb119d04468c09aaeca not found: ID does not exist" Nov 04 03:09:26.076428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.076399 4784 scope.go:122] "RemoveContainer" containerID="cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3" Nov 04 03:09:26.087635 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.087615 4784 scope.go:122] "RemoveContainer" containerID="cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3" Nov 04 03:09:26.087957 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:26.087934 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3\": container with ID starting with cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3 not found: ID does not exist" containerID="cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3" Nov 04 03:09:26.088008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.087967 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3"} err="failed to get container status \"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3\": rpc error: code = NotFound desc = could not find container \"cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3\": container with ID starting with cb2dd7166628ab50770f55fd8492cac073a558e55ed7f24f10dd30ad63dfe8e3 not found: ID does not exist" Nov 04 03:09:26.222354 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.222319 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 03:09:26.226717 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.226686 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/distinct-w5-3b7476-kserve-6df48d8b99-vnxxm"] Nov 04 03:09:26.337419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.337314 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 03:09:26.341276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:26.341245 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3b7476/simulator-w5-3b7476-kserve-799f77bb97-tcpgt"] Nov 04 03:09:27.162301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:27.162267 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a23946-92d8-4c17-ac49-799f536621a5" path="/var/lib/kubelet/pods/15a23946-92d8-4c17-ac49-799f536621a5/volumes" Nov 04 03:09:27.162784 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:27.162770 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" path="/var/lib/kubelet/pods/8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02/volumes" Nov 04 03:09:32.145212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:32.144592 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:09:33.512218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.511359 4784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Nov 04 03:09:33.663357 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.663300 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.698223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.698130 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Nov 04 03:09:33.699436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.699394 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.723988 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.723902 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.738150 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:33.737306 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:36.193202 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:36.193155 4784 scope.go:122] "RemoveContainer" containerID="4a7b9921b0129fd9ad44b287fd5b0cdfc34e39b1cd5f9423dfd766d42af04b0d" Nov 04 03:09:36.402806 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:36.402766 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:09:42.144116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:42.144076 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:09:44.320142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.320112 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 03:09:44.345876 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.345840 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\" (UniqueName: \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\") pod \"9246b627-cbd1-4314-9e93-f3b843db317d\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " Nov 04 03:09:44.346043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.345970 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\") pod \"9246b627-cbd1-4314-9e93-f3b843db317d\" (UID: \"9246b627-cbd1-4314-9e93-f3b843db317d\") " Nov 04 03:09:44.346373 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.346347 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume" pod "9246b627-cbd1-4314-9e93-f3b843db317d" (UID: "9246b627-cbd1-4314-9e93-f3b843db317d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:44.346608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.346590 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9246b627-cbd1-4314-9e93-f3b843db317d-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:44.348251 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.348229 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4" pod "9246b627-cbd1-4314-9e93-f3b843db317d" (UID: "9246b627-cbd1-4314-9e93-f3b843db317d"). InnerVolumeSpecName "kube-api-access-4vsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:44.431974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.431935 4784 generic.go:356] "Generic (PLEG): container finished" podID="9246b627-cbd1-4314-9e93-f3b843db317d" containerID="1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec" exitCode=0 Nov 04 03:09:44.432357 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.431987 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" event={"ID":"9246b627-cbd1-4314-9e93-f3b843db317d","Type":"ContainerDied","Data":"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec"} Nov 04 03:09:44.432505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.432006 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" Nov 04 03:09:44.432505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.432394 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b" event={"ID":"9246b627-cbd1-4314-9e93-f3b843db317d","Type":"ContainerDied","Data":"d9d6d81c5d16bfb995180bbfee14ca62ed7896727d86ff2e4f756b66cadcaff0"} Nov 04 03:09:44.432505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.432428 4784 scope.go:122] "RemoveContainer" containerID="1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec" Nov 04 03:09:44.447684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.447652 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsq4\" (UniqueName: \"kubernetes.io/projected/9246b627-cbd1-4314-9e93-f3b843db317d-kube-api-access-4vsq4\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:44.447967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.447893 4784 scope.go:122] "RemoveContainer" containerID="1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec" Nov 04 03:09:44.448156 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:44.448128 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec\": container with ID starting with 1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec not found: ID does not exist" containerID="1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec" Nov 04 03:09:44.448240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.448168 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec"} err="failed to get container status \"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec\": rpc error: code = NotFound desc = could not find container \"1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec\": container with ID starting with 1c7c58f2ad16e66196b1923a67446b53c3fa7df2462e7934e26a55b6bf4b36ec not found: ID does not exist" Nov 04 03:09:44.613826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.613729 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 03:09:44.617966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:44.617936 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b7476-56c5fdb86c-8786b"] Nov 04 03:09:45.161782 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:45.161748 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" path="/var/lib/kubelet/pods/9246b627-cbd1-4314-9e93-f3b843db317d/volumes" Nov 04 03:09:51.676380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:51.676335 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:09:51.676921 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:51.676562 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" containerID="cri-o://7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5" gracePeriod=30 Nov 04 03:09:54.521773 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:54.521731 4784 patch_prober.go:29] interesting pod/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:09:54.522260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:54.521788 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:09:55.759978 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:55.759941 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx"] Nov 04 03:09:55.763996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:55.763958 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57f116-7nftx"] Nov 04 03:09:56.960002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:56.959929 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:09:57.057885 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.057849 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.057885 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.057891 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.057971 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058010 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058121 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058227 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058263 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:57.058429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058313 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:57.058492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058420 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:57.058492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058447 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:57.058609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058587 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058687 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.058774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.058757 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\") pod \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\" (UID: \"8dcdf837-d0c4-4a46-a9eb-caf3ea268789\") " Nov 04 03:09:57.059262 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059232 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:57.059723 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059696 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.059856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059729 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.059856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059746 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.059856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059761 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.059856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.059775 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.060735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.060714 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:57.060874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.060758 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:57.061133 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.061107 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:57.061243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.061161 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7" pod "8dcdf837-d0c4-4a46-a9eb-caf3ea268789" (UID: "8dcdf837-d0c4-4a46-a9eb-caf3ea268789"). InnerVolumeSpecName "kube-api-access-h79l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:57.161040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.161000 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.161040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.161043 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.161314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.161058 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.161314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.161075 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79l7\" (UniqueName: \"kubernetes.io/projected/8dcdf837-d0c4-4a46-a9eb-caf3ea268789-kube-api-access-h79l7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:57.165074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.165043 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2" path="/var/lib/kubelet/pods/cf8cfc85-d16d-4553-b5d7-d4a4c21b71e2/volumes" Nov 04 03:09:57.446989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.446953 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:09:57.447815 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447790 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:09:57.447815 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447814 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447838 4784 cpu_manager.go:414] "containerMap: removing container" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447847 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447861 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a4799f0b-61e8-4b28-b9b5-29f80077a213" containerName="revoke-keys" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447868 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4799f0b-61e8-4b28-b9b5-29f80077a213" containerName="revoke-keys" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447885 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447892 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447921 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" Nov 04 03:09:57.448037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.447928 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448171 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448205 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a23946-92d8-4c17-ac49-799f536621a5" containerName="main" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448217 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4799f0b-61e8-4b28-b9b5-29f80077a213" containerName="revoke-keys" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448226 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d67afa4-bd4c-4e78-bdd5-ae39c6d45e02" containerName="main" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448238 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:09:57.448456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.448252 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9246b627-cbd1-4314-9e93-f3b843db317d" containerName="payload-pre-processing" Nov 04 03:09:57.453032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.453010 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.538200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.538154 4784 generic.go:356] "Generic (PLEG): container finished" podID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerID="7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5" exitCode=0 Nov 04 03:09:57.538389 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.538225 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerDied","Data":"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5"} Nov 04 03:09:57.538389 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.538247 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" Nov 04 03:09:57.538389 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.538285 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj" event={"ID":"8dcdf837-d0c4-4a46-a9eb-caf3ea268789","Type":"ContainerDied","Data":"2b74170f880a802a3467f8402b4361d040098d5d3fd8063096bc894bd6e0bd40"} Nov 04 03:09:57.538389 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.538316 4784 scope.go:122] "RemoveContainer" containerID="7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5" Nov 04 03:09:57.549704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.549686 4784 scope.go:122] "RemoveContainer" containerID="dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f" Nov 04 03:09:57.560270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.560249 4784 scope.go:122] "RemoveContainer" containerID="7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5" Nov 04 03:09:57.560566 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:57.560543 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5\": container with ID starting with 7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5 not found: ID does not exist" containerID="7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5" Nov 04 03:09:57.560662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.560579 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5"} err="failed to get container status \"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5\": rpc error: code = NotFound desc = could not find container \"7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5\": container with ID starting with 7004f8bb7b8d36bac06f946452b8fe4e4158a37e4724c241be3d27413e0315a5 not found: ID does not exist" Nov 04 03:09:57.560662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.560605 4784 scope.go:122] "RemoveContainer" containerID="dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f" Nov 04 03:09:57.560850 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:09:57.560828 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f\": container with ID starting with dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f not found: ID does not exist" containerID="dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f" Nov 04 03:09:57.560892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.560860 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f"} err="failed to get container status \"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f\": rpc error: code = NotFound desc = could not find container \"dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f\": container with ID starting with dd493951c17d6b964107984247857a0ac9b6be539b6d0646604e96e80408497f not found: ID does not exist" Nov 04 03:09:57.565843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.565812 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5wb\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566009 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.565858 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.565996 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566124 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566157 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566315 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566402 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566448 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.566559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.566493 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669163 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669281 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669321 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669386 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669491 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5wb\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669531 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669567 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669682 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.669759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.669716 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.670559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.670512 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.670821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.670794 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.671348 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.671072 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.671348 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.671083 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:09:57.671537 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.671436 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.671849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.671821 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.674660 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.674635 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.675029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.675007 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.692736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.692697 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:09:57.703265 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.700248 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-569fb7-openshift-default-6dfd58f675-h2ttj"] Nov 04 03:09:57.732467 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.732431 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.733821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.733799 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5wb\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:57.767506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:57.767466 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:58.221515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.221472 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:09:58.222252 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:09:58.222175 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45065a71_f569_4992_9475_ca501ce07054.slice/crio-4bfcea5dd7ebe147aa1eef3a1025e79f3a23a1efbf38df56c0bcab1cd70152b6 WatchSource:0}: Error finding container 4bfcea5dd7ebe147aa1eef3a1025e79f3a23a1efbf38df56c0bcab1cd70152b6: Status 404 returned error can't find the container with id 4bfcea5dd7ebe147aa1eef3a1025e79f3a23a1efbf38df56c0bcab1cd70152b6 Nov 04 03:09:58.226148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.226102 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:09:58.226338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.226221 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:09:58.226338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.226268 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:09:58.547995 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.547888 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" event={"ID":"45065a71-f569-4992-9475-ca501ce07054","Type":"ContainerStarted","Data":"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf"} Nov 04 03:09:58.547995 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.547958 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" event={"ID":"45065a71-f569-4992-9475-ca501ce07054","Type":"ContainerStarted","Data":"4bfcea5dd7ebe147aa1eef3a1025e79f3a23a1efbf38df56c0bcab1cd70152b6"} Nov 04 03:09:58.678949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.678888 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podStartSLOduration=1.678870071 podStartE2EDuration="1.678870071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:58.674110399 +0000 UTC m=+2245.640611391" watchObservedRunningTime="2026-11-04 03:09:58.678870071 +0000 UTC m=+2245.645371064" Nov 04 03:09:58.774214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:58.769296 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:09:59.189860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:59.189822 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" path="/var/lib/kubelet/pods/8dcdf837-d0c4-4a46-a9eb-caf3ea268789/volumes" Nov 04 03:09:59.775231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:59.774006 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:09:59.775231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:59.774071 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:09:59.890685 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:09:59.890644 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:10:00.775695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:00.775654 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:10:00.776363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:00.776330 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:10:01.780855 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:01.780813 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:10:01.781508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:01.780894 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:10:02.772733 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:02.772690 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:10:02.773077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:02.773048 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:10:03.777216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:03.773421 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:10:03.777216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:03.773489 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:10:04.768349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:04.768304 4784 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 04 03:10:04.768552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:04.768380 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 04 03:10:04.891854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:04.891806 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" podUID="4345f149-42a6-4102-9412-3353b0ac2837" containerName="payload-pre-processing" containerID="cri-o://674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f" gracePeriod=25 Nov 04 03:10:05.258872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.258836 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:10:05.341104 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.340993 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\" (UniqueName: \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\") pod \"4345f149-42a6-4102-9412-3353b0ac2837\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " Nov 04 03:10:05.341524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.341497 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\") pod \"4345f149-42a6-4102-9412-3353b0ac2837\" (UID: \"4345f149-42a6-4102-9412-3353b0ac2837\") " Nov 04 03:10:05.342971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.342934 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume" pod "4345f149-42a6-4102-9412-3353b0ac2837" (UID: "4345f149-42a6-4102-9412-3353b0ac2837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:05.347637 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.347606 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc" pod "4345f149-42a6-4102-9412-3353b0ac2837" (UID: "4345f149-42a6-4102-9412-3353b0ac2837"). InnerVolumeSpecName "kube-api-access-fcqkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:05.443875 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.443837 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqkc\" (UniqueName: \"kubernetes.io/projected/4345f149-42a6-4102-9412-3353b0ac2837-kube-api-access-fcqkc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:05.443875 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.443874 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4345f149-42a6-4102-9412-3353b0ac2837-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:05.661549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.661447 4784 generic.go:356] "Generic (PLEG): container finished" podID="4345f149-42a6-4102-9412-3353b0ac2837" containerID="674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f" exitCode=0 Nov 04 03:10:05.661549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.661491 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" Nov 04 03:10:05.661549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.661527 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" event={"ID":"4345f149-42a6-4102-9412-3353b0ac2837","Type":"ContainerDied","Data":"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f"} Nov 04 03:10:05.661889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.661586 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9" event={"ID":"4345f149-42a6-4102-9412-3353b0ac2837","Type":"ContainerDied","Data":"6f80d39d8df5adf18ae138ae409f37c0eb7dfaaf6637d004c6df515954b9ce97"} Nov 04 03:10:05.661889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.661614 4784 scope.go:122] "RemoveContainer" containerID="674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f" Nov 04 03:10:05.677410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.677382 4784 scope.go:122] "RemoveContainer" containerID="674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f" Nov 04 03:10:05.677844 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:05.677820 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f\": container with ID starting with 674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f not found: ID does not exist" containerID="674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f" Nov 04 03:10:05.677965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.677852 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f"} err="failed to get container status \"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f\": rpc error: code = NotFound desc = could not find container \"674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f\": container with ID starting with 674625541cc03c9f0f01ad725aaf4fbf8942a338ca527348eb8c6b03c4d9038f not found: ID does not exist" Nov 04 03:10:05.833120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.833086 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:10:05.929213 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:05.922310 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9d2b96-7b7d78cbff-gtqd9"] Nov 04 03:10:06.174156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:06.174121 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:10:06.174845 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:06.174775 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:10:06.177980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:06.177958 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:10:06.413058 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:06.412956 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:10:07.162011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:07.161968 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4345f149-42a6-4102-9412-3353b0ac2837" path="/var/lib/kubelet/pods/4345f149-42a6-4102-9412-3353b0ac2837/volumes" Nov 04 03:10:18.088728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:18.088686 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:10:18.089173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:18.088945 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" containerID="cri-o://669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337" gracePeriod=30 Nov 04 03:10:21.308965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:21.308928 4784 patch_prober.go:29] interesting pod/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:10:21.309445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:21.308998 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:10:23.426995 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.426968 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:10:23.525889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.525857 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.525889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.525903 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526002 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526028 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526125 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526078 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526152 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526177 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:23.526342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526241 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:23.526342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526322 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:23.526478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526376 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526431 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:23.526478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526448 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526580 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526495 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\") pod \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\" (UID: \"08ebf2e9-1003-4ae4-8710-bcabe3592dd0\") " Nov 04 03:10:23.526674 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.526653 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:23.527082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.527059 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.527154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.527086 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.527154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.527097 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.527154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.527106 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.527154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.527114 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.528760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.528733 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:23.529077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.529052 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:23.529210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.529107 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:23.529289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.529264 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7" pod "08ebf2e9-1003-4ae4-8710-bcabe3592dd0" (UID: "08ebf2e9-1003-4ae4-8710-bcabe3592dd0"). InnerVolumeSpecName "kube-api-access-gvpc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:23.628931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.628889 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.628931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.628926 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpc7\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-kube-api-access-gvpc7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.629174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.628945 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.629174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.628954 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ebf2e9-1003-4ae4-8710-bcabe3592dd0-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:23.828269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.828234 4784 generic.go:356] "Generic (PLEG): container finished" podID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerID="669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337" exitCode=0 Nov 04 03:10:23.828509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.828303 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerDied","Data":"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337"} Nov 04 03:10:23.828509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.828353 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" Nov 04 03:10:23.828509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.828382 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5" event={"ID":"08ebf2e9-1003-4ae4-8710-bcabe3592dd0","Type":"ContainerDied","Data":"371bf23f8e8aa5893c6f296045ec5fc0e49bacf98b49ce6572e609917de4fc0f"} Nov 04 03:10:23.828509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.828406 4784 scope.go:122] "RemoveContainer" containerID="669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337" Nov 04 03:10:23.840452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.840423 4784 scope.go:122] "RemoveContainer" containerID="a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0" Nov 04 03:10:23.851633 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.851611 4784 scope.go:122] "RemoveContainer" containerID="669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337" Nov 04 03:10:23.851963 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:23.851939 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337\": container with ID starting with 669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337 not found: ID does not exist" containerID="669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337" Nov 04 03:10:23.852050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.851972 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337"} err="failed to get container status \"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337\": rpc error: code = NotFound desc = could not find container \"669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337\": container with ID starting with 669bc5ac1cd0f7e033c3dded3cd83ee3a1a625038c71e11b48ca5bebcafc5337 not found: ID does not exist" Nov 04 03:10:23.852050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.851992 4784 scope.go:122] "RemoveContainer" containerID="a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0" Nov 04 03:10:23.852326 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:23.852302 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0\": container with ID starting with a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0 not found: ID does not exist" containerID="a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0" Nov 04 03:10:23.852408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:23.852341 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0"} err="failed to get container status \"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0\": rpc error: code = NotFound desc = could not find container \"a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0\": container with ID starting with a68a7853ba4d83dceb83bb1d046f3896626d32b5e891a7a600047f53ca9999b0 not found: ID does not exist" Nov 04 03:10:24.001808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:24.001769 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:10:24.008723 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:24.008693 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-00c846-openshift-default-785fd9fcdc-kf4s5"] Nov 04 03:10:25.187743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:25.187705 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" path="/var/lib/kubelet/pods/08ebf2e9-1003-4ae4-8710-bcabe3592dd0/volumes" Nov 04 03:10:26.542323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:26.542289 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:10:26.542901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:26.542510 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" containerID="cri-o://83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf" gracePeriod=30 Nov 04 03:10:26.546971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:26.546945 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 03:10:26.547223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:26.547170 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" containerID="cri-o://ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09" gracePeriod=30 Nov 04 03:10:28.852123 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.852073 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:10:28.853089 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853068 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4345f149-42a6-4102-9412-3353b0ac2837" containerName="payload-pre-processing" Nov 04 03:10:28.853136 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853094 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4345f149-42a6-4102-9412-3353b0ac2837" containerName="payload-pre-processing" Nov 04 03:10:28.853172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853142 4784 cpu_manager.go:414] "containerMap: removing container" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.853172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853151 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.853263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853181 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:10:28.853263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853205 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcdf837-d0c4-4a46-a9eb-caf3ea268789" containerName="istio-proxy" Nov 04 03:10:28.853263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853218 4784 cpu_manager.go:414] "containerMap: removing container" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.853263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853226 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.853414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853401 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4345f149-42a6-4102-9412-3353b0ac2837" containerName="payload-pre-processing" Nov 04 03:10:28.853453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853426 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.853453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.853439 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ebf2e9-1003-4ae4-8710-bcabe3592dd0" containerName="istio-proxy" Nov 04 03:10:28.858528 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.858502 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:28.956251 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.955120 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:10:28.966579 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.966377 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cbcf738\"" Nov 04 03:10:28.966579 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:28.966540 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cbcf738-dockercfg-pnsst\"" Nov 04 03:10:29.046364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.046304 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.046364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.046370 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvk9\" (UniqueName: \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.147514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.147420 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.147514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.147468 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvk9\" (UniqueName: \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.148390 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.148359 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.212005 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.211974 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvk9\" (UniqueName: \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.473899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.473857 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:29.865649 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:10:29.865601 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45de264f_b1d0_4425_9b7f_dd4a8d0df5c0.slice/crio-85f960803480eeb2060de917fe3b37ef413f1436065daf6d33ae1d7818a36a10 WatchSource:0}: Error finding container 85f960803480eeb2060de917fe3b37ef413f1436065daf6d33ae1d7818a36a10: Status 404 returned error can't find the container with id 85f960803480eeb2060de917fe3b37ef413f1436065daf6d33ae1d7818a36a10 Nov 04 03:10:29.866098 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.865800 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:10:29.905802 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:29.905759 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" event={"ID":"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0","Type":"ContainerStarted","Data":"85f960803480eeb2060de917fe3b37ef413f1436065daf6d33ae1d7818a36a10"} Nov 04 03:10:30.915019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:30.914978 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" event={"ID":"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0","Type":"ContainerStarted","Data":"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd"} Nov 04 03:10:30.915484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:30.915049 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:31.034733 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.034670 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" podStartSLOduration=3.034654621 podStartE2EDuration="3.034654621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:10:31.033578405 +0000 UTC m=+2278.000079398" watchObservedRunningTime="2026-11-04 03:10:31.034654621 +0000 UTC m=+2278.001155653" Nov 04 03:10:31.884393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.884365 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:10:31.903276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903242 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903311 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903362 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903440 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903527 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903561 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903609 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.903759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903643 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903682 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.903915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.903769 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\") pod \"45065a71-f569-4992-9475-ca501ce07054\" (UID: \"45065a71-f569-4992-9475-ca501ce07054\") " Nov 04 03:10:31.904043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904022 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.904114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904059 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.904328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904309 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.904565 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904541 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.904657 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904566 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.904657 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904580 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.904657 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904594 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.904930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.904824 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:31.906495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.906456 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "kube-api-access-pj5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:31.906592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.906548 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:31.906793 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.906776 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.906844 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.906795 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo" pod "45065a71-f569-4992-9475-ca501ce07054" (UID: "45065a71-f569-4992-9475-ca501ce07054"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:31.918341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.918316 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 03:10:31.924630 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.924596 4784 generic.go:356] "Generic (PLEG): container finished" podID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerID="ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09" exitCode=0 Nov 04 03:10:31.924805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.924698 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" Nov 04 03:10:31.924805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.924731 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" event={"ID":"47dd8e0e-f14c-4b80-93ed-569fc70494e4","Type":"ContainerDied","Data":"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09"} Nov 04 03:10:31.924911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.924838 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr" event={"ID":"47dd8e0e-f14c-4b80-93ed-569fc70494e4","Type":"ContainerDied","Data":"473a5299099373ec1b4adb4295fc71c671e9610065b4bd9a1fe186ab659e0d53"} Nov 04 03:10:31.924911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.924868 4784 scope.go:122] "RemoveContainer" containerID="ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09" Nov 04 03:10:31.926154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.926130 4784 generic.go:356] "Generic (PLEG): container finished" podID="45065a71-f569-4992-9475-ca501ce07054" containerID="83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf" exitCode=0 Nov 04 03:10:31.926269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.926182 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" Nov 04 03:10:31.926269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.926221 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" event={"ID":"45065a71-f569-4992-9475-ca501ce07054","Type":"ContainerDied","Data":"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf"} Nov 04 03:10:31.926269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.926265 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp" event={"ID":"45065a71-f569-4992-9475-ca501ce07054","Type":"ContainerDied","Data":"4bfcea5dd7ebe147aa1eef3a1025e79f3a23a1efbf38df56c0bcab1cd70152b6"} Nov 04 03:10:31.935981 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.935955 4784 scope.go:122] "RemoveContainer" containerID="ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09" Nov 04 03:10:31.936367 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:31.936345 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09\": container with ID starting with ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09 not found: ID does not exist" containerID="ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09" Nov 04 03:10:31.936451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.936377 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09"} err="failed to get container status \"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09\": rpc error: code = NotFound desc = could not find container \"ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09\": container with ID starting with ff3766c31feef1abd0005e2cf05a004eccfd9919cd2ff3e6ba351a9c3fc1cc09 not found: ID does not exist" Nov 04 03:10:31.936451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.936397 4784 scope.go:122] "RemoveContainer" containerID="83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf" Nov 04 03:10:31.948021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.947997 4784 scope.go:122] "RemoveContainer" containerID="83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf" Nov 04 03:10:31.948463 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:31.948445 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf\": container with ID starting with 83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf not found: ID does not exist" containerID="83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf" Nov 04 03:10:31.948527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:31.948473 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf"} err="failed to get container status \"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf\": rpc error: code = NotFound desc = could not find container \"83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf\": container with ID starting with 83aac73176ec30bc40abe11eb8917c9aea792de27b075717206a3bbd49b8d5cf not found: ID does not exist" Nov 04 03:10:32.005904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.005869 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.005916 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.005959 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006003 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006104 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006243 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:32.006358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006261 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006471 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006585 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006527 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:32.006585 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006542 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006624 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\") pod \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\" (UID: \"47dd8e0e-f14c-4b80-93ed-569fc70494e4\") " Nov 04 03:10:32.006764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.006712 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:32.007274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007248 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007272 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007288 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45065a71-f569-4992-9475-ca501ce07054-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007299 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45065a71-f569-4992-9475-ca501ce07054-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007309 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj5wb\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-kube-api-access-pj5wb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007319 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45065a71-f569-4992-9475-ca501ce07054-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007328 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45065a71-f569-4992-9475-ca501ce07054-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007336 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.007847 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007672 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:32.007847 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.007596 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:32.009967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.009941 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:32.010074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.009983 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:32.010074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.010009 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:32.010074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.010039 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw" pod "47dd8e0e-f14c-4b80-93ed-569fc70494e4" (UID: "47dd8e0e-f14c-4b80-93ed-569fc70494e4"). InnerVolumeSpecName "kube-api-access-j5jvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108333 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108371 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108381 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108390 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd8e0e-f14c-4b80-93ed-569fc70494e4-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108401 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd8e0e-f14c-4b80-93ed-569fc70494e4-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.108429 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.108411 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jvw\" (UniqueName: \"kubernetes.io/projected/47dd8e0e-f14c-4b80-93ed-569fc70494e4-kube-api-access-j5jvw\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.306177 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.306137 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:10:32.310530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.310500 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-7zmxp"] Nov 04 03:10:32.424006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.423970 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 03:10:32.430355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.430319 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-6n8hr"] Nov 04 03:10:32.938510 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.938478 4784 generic.go:356] "Generic (PLEG): container finished" podID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerID="1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4" exitCode=0 Nov 04 03:10:32.939000 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.938554 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerDied","Data":"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4"} Nov 04 03:10:32.939046 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.939001 4784 scope.go:122] "RemoveContainer" containerID="1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4" Nov 04 03:10:32.940672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.940639 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:10:32.940804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.940733 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:10:32.940804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:32.940781 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:10:33.162687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.162656 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45065a71-f569-4992-9475-ca501ce07054" path="/var/lib/kubelet/pods/45065a71-f569-4992-9475-ca501ce07054/volumes" Nov 04 03:10:33.163168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.163154 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" path="/var/lib/kubelet/pods/47dd8e0e-f14c-4b80-93ed-569fc70494e4/volumes" Nov 04 03:10:33.361301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.361207 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz"] Nov 04 03:10:33.362022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.361998 4784 cpu_manager.go:414] "containerMap: removing container" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" Nov 04 03:10:33.362022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.362022 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" Nov 04 03:10:33.362278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.362042 4784 cpu_manager.go:414] "containerMap: removing container" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" Nov 04 03:10:33.362278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.362048 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" Nov 04 03:10:33.362278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.362227 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dd8e0e-f14c-4b80-93ed-569fc70494e4" containerName="istio-proxy" Nov 04 03:10:33.362278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.362246 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="45065a71-f569-4992-9475-ca501ce07054" containerName="istio-proxy" Nov 04 03:10:33.365418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.365394 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.418886 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.418850 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:10:33.419091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.419066 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:10:33.420103 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.420061 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz"] Nov 04 03:10:33.430401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.424949 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x285m\" (UniqueName: \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.430401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.425136 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.430401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.425174 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.462363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.462319 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:10:33.462563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.462379 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:10:33.526334 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.526292 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.526334 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.526337 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.526714 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.526689 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x285m\" (UniqueName: \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.527237 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.527209 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.529346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.529326 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.589233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.589181 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x285m\" (UniqueName: \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\") pod \"maas-api-revoke-sub-f65a27aa-68cvz\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.686220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.682437 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:33.948246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:33.948126 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerStarted","Data":"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531"} Nov 04 03:10:34.094091 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:10:34.094047 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8860e9c6_8212_4eff_a6cf_228624c4fd3c.slice/crio-889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f WatchSource:0}: Error finding container 889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f: Status 404 returned error can't find the container with id 889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f Nov 04 03:10:34.129881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:34.129841 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz"] Nov 04 03:10:34.520172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:34.520132 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:10:34.521668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:34.521641 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 04 03:10:34.521818 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:34.521691 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 04 03:10:35.045888 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:35.045818 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" event={"ID":"8860e9c6-8212-4eff-a6cf-228624c4fd3c","Type":"ContainerStarted","Data":"cd237ddcd9e61d305e47c84901743ae3e6283bf5032e1dcac73ff774830fe46b"} Nov 04 03:10:35.046906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:35.046869 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" event={"ID":"8860e9c6-8212-4eff-a6cf-228624c4fd3c","Type":"ContainerStarted","Data":"889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f"} Nov 04 03:10:35.189095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:35.167319 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" podStartSLOduration=2.167298416 podStartE2EDuration="2.167298416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:10:35.166596763 +0000 UTC m=+2282.133097758" watchObservedRunningTime="2026-11-04 03:10:35.167298416 +0000 UTC m=+2282.133799420" Nov 04 03:10:35.519804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:35.519765 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 04 03:10:35.519989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:35.519838 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 04 03:10:36.052943 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:36.052905 4784 generic.go:356] "Generic (PLEG): container finished" podID="8860e9c6-8212-4eff-a6cf-228624c4fd3c" containerID="cd237ddcd9e61d305e47c84901743ae3e6283bf5032e1dcac73ff774830fe46b" exitCode=0 Nov 04 03:10:36.053458 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:36.052997 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" event={"ID":"8860e9c6-8212-4eff-a6cf-228624c4fd3c","Type":"ContainerDied","Data":"cd237ddcd9e61d305e47c84901743ae3e6283bf5032e1dcac73ff774830fe46b"} Nov 04 03:10:36.411543 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:10:36.411504 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:10:36.520459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:36.520418 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 04 03:10:36.520665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:36.520512 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 04 03:10:37.287748 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.287718 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:37.417593 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.417537 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\") pod \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " Nov 04 03:10:37.417779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.417676 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\" (UniqueName: \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\") pod \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " Nov 04 03:10:37.417868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.417812 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\") pod \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\" (UID: \"8860e9c6-8212-4eff-a6cf-228624c4fd3c\") " Nov 04 03:10:37.418412 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.418379 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca" pod "8860e9c6-8212-4eff-a6cf-228624c4fd3c" (UID: "8860e9c6-8212-4eff-a6cf-228624c4fd3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:37.421745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.421719 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m" pod "8860e9c6-8212-4eff-a6cf-228624c4fd3c" (UID: "8860e9c6-8212-4eff-a6cf-228624c4fd3c"). InnerVolumeSpecName "kube-api-access-x285m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:37.422076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.422049 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca" pod "8860e9c6-8212-4eff-a6cf-228624c4fd3c" (UID: "8860e9c6-8212-4eff-a6cf-228624c4fd3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:37.424108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.423069 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:10:37.520274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.520168 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:37.520538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.520514 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x285m\" (UniqueName: \"kubernetes.io/projected/8860e9c6-8212-4eff-a6cf-228624c4fd3c-kube-api-access-x285m\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:37.520538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.520538 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8860e9c6-8212-4eff-a6cf-228624c4fd3c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:37.520759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.520388 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 04 03:10:37.520759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:37.520587 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 04 03:10:38.093156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:38.093115 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" Nov 04 03:10:38.093156 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:38.093137 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz" event={"ID":"8860e9c6-8212-4eff-a6cf-228624c4fd3c","Type":"ContainerDied","Data":"889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f"} Nov 04 03:10:38.093407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:38.093180 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="889f9f7c1f44ea13b4bff95a77ea8ed207d060f40a8fd3cd2598c194da22de1f" Nov 04 03:10:38.519832 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:38.519797 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 04 03:10:38.520262 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:38.519857 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 04 03:10:39.526396 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:39.526364 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:10:39.527956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:39.527936 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:10:39.756183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:39.756144 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:10:39.756701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:39.756667 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" containerID="cri-o://655124c487d1ced04bb29744d3b26da2abefc858ddd83fd6aaf5a6a446076cf1" gracePeriod=30 Nov 04 03:10:41.930270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:41.930233 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:10:42.470760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.470708 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6"] Nov 04 03:10:42.471495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.471471 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8860e9c6-8212-4eff-a6cf-228624c4fd3c" containerName="revoke-keys" Nov 04 03:10:42.471495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.471490 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8860e9c6-8212-4eff-a6cf-228624c4fd3c" containerName="revoke-keys" Nov 04 03:10:42.471709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.471622 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8860e9c6-8212-4eff-a6cf-228624c4fd3c" containerName="revoke-keys" Nov 04 03:10:42.474414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.474391 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.526668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.526576 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.526668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.526656 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftdb\" (UniqueName: \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.527011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.526965 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.528139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.528111 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:10:42.531822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.531794 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6"] Nov 04 03:10:42.578032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.577991 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:10:42.578032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.577991 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:10:42.578288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.578045 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:10:42.627901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.627863 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.628105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.627971 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.628105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.628046 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftdb\" (UniqueName: \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.628652 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.628617 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.630667 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.630646 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.690460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.690418 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftdb\" (UniqueName: \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\") pod \"maas-api-revoke-sub-31f5ca53-8dpd6\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:42.788182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:42.788084 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:43.169621 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:10:43.169588 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b37ae8d_f677_4777_a4f8_1ef21992c30f.slice/crio-ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec WatchSource:0}: Error finding container ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec: Status 404 returned error can't find the container with id ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec Nov 04 03:10:43.170074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:43.170032 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6"] Nov 04 03:10:44.141904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.141865 4784 generic.go:356] "Generic (PLEG): container finished" podID="7b37ae8d-f677-4777-a4f8-1ef21992c30f" containerID="1302c3f71c3cf57ba263e529542bc39cd6b5df315186ab64cb2527fa9ef6a38d" exitCode=0 Nov 04 03:10:44.142096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.141944 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" event={"ID":"7b37ae8d-f677-4777-a4f8-1ef21992c30f","Type":"ContainerDied","Data":"1302c3f71c3cf57ba263e529542bc39cd6b5df315186ab64cb2527fa9ef6a38d"} Nov 04 03:10:44.142096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.141999 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" event={"ID":"7b37ae8d-f677-4777-a4f8-1ef21992c30f","Type":"ContainerStarted","Data":"ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec"} Nov 04 03:10:44.722555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.722514 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s"] Nov 04 03:10:44.726003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.725977 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.788172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.788133 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s"] Nov 04 03:10:44.862099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.862055 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.862330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.862215 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.862330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.862264 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbw26\" (UniqueName: \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.964152 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.964121 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.964297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.964161 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbw26\" (UniqueName: \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.964463 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.964437 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.965579 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.965533 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:44.967518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:44.967495 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:45.027148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.027112 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbw26\" (UniqueName: \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\") pod \"maas-api-revoke-sub-db0a837b-r245s\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:45.040084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.040054 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:45.161289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.161257 4784 generic.go:356] "Generic (PLEG): container finished" podID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerID="655124c487d1ced04bb29744d3b26da2abefc858ddd83fd6aaf5a6a446076cf1" exitCode=0 Nov 04 03:10:45.168853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.168803 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerDied","Data":"655124c487d1ced04bb29744d3b26da2abefc858ddd83fd6aaf5a6a446076cf1"} Nov 04 03:10:45.168990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.168911 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" event={"ID":"2afd3c41-3375-4ae7-8e07-e18676b0c3ec","Type":"ContainerDied","Data":"a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62"} Nov 04 03:10:45.168990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.168943 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a30a8155561acf77ae6ec238ea7eb0be78bf346cb2291376dcc151a58f8a62" Nov 04 03:10:45.168990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.168963 4784 scope.go:122] "RemoveContainer" containerID="ccc80c34705355278a817cc9f4c77051f872faf20d0c6592b0cf621165ac004a" Nov 04 03:10:45.170478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.170459 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:10:45.274839 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.274799 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.274871 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.274981 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275098 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275154 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275157 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:45.275361 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275241 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275361 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275275 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:45.275480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275364 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275406 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275486 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\") pod \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\" (UID: \"2afd3c41-3375-4ae7-8e07-e18676b0c3ec\") " Nov 04 03:10:45.275588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275534 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:45.275832 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.275808 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:45.276347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.276298 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.276347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.276322 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.276347 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.276338 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.276703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.276354 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.276972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.276936 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:45.278545 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.278338 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:45.280157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.280129 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "kube-api-access-nl5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:45.280314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.280246 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:45.280865 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.280840 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo" pod "2afd3c41-3375-4ae7-8e07-e18676b0c3ec" (UID: "2afd3c41-3375-4ae7-8e07-e18676b0c3ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:45.336753 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.336719 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:45.377274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.377233 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.377274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.377264 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.377274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.377275 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5q9\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-kube-api-access-nl5q9\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.377538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.377287 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.377538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.377296 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd3c41-3375-4ae7-8e07-e18676b0c3ec-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.464970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.464931 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s"] Nov 04 03:10:45.478749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.478714 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\") pod \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " Nov 04 03:10:45.478942 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.478927 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\") pod \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " Nov 04 03:10:45.479122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.479090 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca" pod "7b37ae8d-f677-4777-a4f8-1ef21992c30f" (UID: "7b37ae8d-f677-4777-a4f8-1ef21992c30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:45.479122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.479094 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\" (UniqueName: \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\") pod \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\" (UID: \"7b37ae8d-f677-4777-a4f8-1ef21992c30f\") " Nov 04 03:10:45.479628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.479608 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.481503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.481477 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca" pod "7b37ae8d-f677-4777-a4f8-1ef21992c30f" (UID: "7b37ae8d-f677-4777-a4f8-1ef21992c30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:45.481633 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.481609 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb" pod "7b37ae8d-f677-4777-a4f8-1ef21992c30f" (UID: "7b37ae8d-f677-4777-a4f8-1ef21992c30f"). InnerVolumeSpecName "kube-api-access-5ftdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:45.500763 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:10:45.500736 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbeb4fc_1114_4c38_88a6_d5b404d2d46b.slice/crio-4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1 WatchSource:0}: Error finding container 4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1: Status 404 returned error can't find the container with id 4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1 Nov 04 03:10:45.581149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.581105 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ftdb\" (UniqueName: \"kubernetes.io/projected/7b37ae8d-f677-4777-a4f8-1ef21992c30f-kube-api-access-5ftdb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:45.581149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:45.581143 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b37ae8d-f677-4777-a4f8-1ef21992c30f-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:46.170048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.170024 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4" Nov 04 03:10:46.171996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.171965 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" event={"ID":"efbeb4fc-1114-4c38-88a6-d5b404d2d46b","Type":"ContainerStarted","Data":"a3dc056bea0932df588fad807083b4b3b120070c9044d2f5a1e0df317fd3d655"} Nov 04 03:10:46.172142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.172018 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" event={"ID":"efbeb4fc-1114-4c38-88a6-d5b404d2d46b","Type":"ContainerStarted","Data":"4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1"} Nov 04 03:10:46.173559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.173535 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" Nov 04 03:10:46.173686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.173569 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6" event={"ID":"7b37ae8d-f677-4777-a4f8-1ef21992c30f","Type":"ContainerDied","Data":"ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec"} Nov 04 03:10:46.173686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.173611 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed02e3199c2b012feec3651162b2e61e3bfab1048cb111df0d81b4351f6638ec" Nov 04 03:10:46.479732 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.479592 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" podStartSLOduration=2.479570955 podStartE2EDuration="2.479570955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:10:46.473337035 +0000 UTC m=+2293.439838026" watchObservedRunningTime="2026-11-04 03:10:46.479570955 +0000 UTC m=+2293.446071948" Nov 04 03:10:46.591150 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.591111 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:10:46.595729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.595699 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9f4199-openshift-default-586448895-hl8h4"] Nov 04 03:10:46.785270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.785171 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 03:10:46.796811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.796777 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 03:10:46.804088 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:46.804058 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 03:10:47.170953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:47.170904 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" path="/var/lib/kubelet/pods/2afd3c41-3375-4ae7-8e07-e18676b0c3ec/volumes" Nov 04 03:10:47.185911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:47.185867 4784 generic.go:356] "Generic (PLEG): container finished" podID="efbeb4fc-1114-4c38-88a6-d5b404d2d46b" containerID="a3dc056bea0932df588fad807083b4b3b120070c9044d2f5a1e0df317fd3d655" exitCode=0 Nov 04 03:10:47.186102 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:47.185961 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" event={"ID":"efbeb4fc-1114-4c38-88a6-d5b404d2d46b","Type":"ContainerDied","Data":"a3dc056bea0932df588fad807083b4b3b120070c9044d2f5a1e0df317fd3d655"} Nov 04 03:10:48.354372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.354344 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:48.527005 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.526918 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\") pod \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " Nov 04 03:10:48.527172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.527051 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\" (UniqueName: \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\") pod \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " Nov 04 03:10:48.527172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.527123 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\") pod \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\" (UID: \"efbeb4fc-1114-4c38-88a6-d5b404d2d46b\") " Nov 04 03:10:48.527489 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.527465 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca" pod "efbeb4fc-1114-4c38-88a6-d5b404d2d46b" (UID: "efbeb4fc-1114-4c38-88a6-d5b404d2d46b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:48.527736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.527717 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:48.529214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.529163 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca" pod "efbeb4fc-1114-4c38-88a6-d5b404d2d46b" (UID: "efbeb4fc-1114-4c38-88a6-d5b404d2d46b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:48.529328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.529308 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26" pod "efbeb4fc-1114-4c38-88a6-d5b404d2d46b" (UID: "efbeb4fc-1114-4c38-88a6-d5b404d2d46b"). InnerVolumeSpecName "kube-api-access-zbw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:48.629142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.629105 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:48.629142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:48.629138 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbw26\" (UniqueName: \"kubernetes.io/projected/efbeb4fc-1114-4c38-88a6-d5b404d2d46b-kube-api-access-zbw26\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:49.098410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:49.098362 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 03:10:49.203243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:49.203201 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" event={"ID":"efbeb4fc-1114-4c38-88a6-d5b404d2d46b","Type":"ContainerDied","Data":"4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1"} Nov 04 03:10:49.203431 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:49.203263 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d28bf12a7101cd6fb194fe292769ff3fcf96066e3a7d7730fa9d3cf3b6ae1f1" Nov 04 03:10:49.203431 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:49.203271 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s" Nov 04 03:10:58.987591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.987548 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf"] Nov 04 03:10:58.988174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988143 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988174 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988206 4784 cpu_manager.go:414] "containerMap: removing container" podUID="7b37ae8d-f677-4777-a4f8-1ef21992c30f" containerName="revoke-keys" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988212 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b37ae8d-f677-4777-a4f8-1ef21992c30f" containerName="revoke-keys" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988226 4784 cpu_manager.go:414] "containerMap: removing container" podUID="efbeb4fc-1114-4c38-88a6-d5b404d2d46b" containerName="revoke-keys" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988232 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbeb4fc-1114-4c38-88a6-d5b404d2d46b" containerName="revoke-keys" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988252 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:10:58.988294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988257 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:10:58.988548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988370 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:10:58.988548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988380 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b37ae8d-f677-4777-a4f8-1ef21992c30f" containerName="revoke-keys" Nov 04 03:10:58.988548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.988387 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbeb4fc-1114-4c38-88a6-d5b404d2d46b" containerName="revoke-keys" Nov 04 03:10:58.991775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:58.991757 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.049632 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.049596 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf"] Nov 04 03:10:59.087022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.086976 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.087228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.087053 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zf7z\" (UniqueName: \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.087228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.087100 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.095665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.095632 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:10:59.096631 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.096605 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:10:59.096735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.096655 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:10:59.096735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.096701 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:10:59.188062 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.188029 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.188338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.188119 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zf7z\" (UniqueName: \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.188338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.188148 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.188955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.188866 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.190636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.190611 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.250901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.250829 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zf7z\" (UniqueName: \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\") pod \"maas-api-revoke-sub-9fedd328-9ttjf\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.306077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.306044 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:10:59.689101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:10:59.689065 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf"] Nov 04 03:10:59.689413 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:10:59.689392 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0129b8_9f5c_4278_a80f_d5068550ffa9.slice/crio-2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10 WatchSource:0}: Error finding container 2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10: Status 404 returned error can't find the container with id 2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10 Nov 04 03:11:00.284027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:00.283930 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" event={"ID":"ad0129b8-9f5c-4278-a80f-d5068550ffa9","Type":"ContainerStarted","Data":"ebbde593ef0ae571fdbc673f5ed09e12bc3022d2e656eecb08c49784620fcf71"} Nov 04 03:11:00.284027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:00.283987 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" event={"ID":"ad0129b8-9f5c-4278-a80f-d5068550ffa9","Type":"ContainerStarted","Data":"2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10"} Nov 04 03:11:00.339915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:00.339114 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 03:11:00.452548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:00.452488 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" podStartSLOduration=2.452474231 podStartE2EDuration="2.452474231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:00.449386422 +0000 UTC m=+2307.415887413" watchObservedRunningTime="2026-11-04 03:11:00.452474231 +0000 UTC m=+2307.418975224" Nov 04 03:11:01.293337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.293298 4784 generic.go:356] "Generic (PLEG): container finished" podID="ad0129b8-9f5c-4278-a80f-d5068550ffa9" containerID="ebbde593ef0ae571fdbc673f5ed09e12bc3022d2e656eecb08c49784620fcf71" exitCode=0 Nov 04 03:11:01.293761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.293380 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" event={"ID":"ad0129b8-9f5c-4278-a80f-d5068550ffa9","Type":"ContainerDied","Data":"ebbde593ef0ae571fdbc673f5ed09e12bc3022d2e656eecb08c49784620fcf71"} Nov 04 03:11:01.808529 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.808468 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" containerID="cri-o://c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09" gracePeriod=45 Nov 04 03:11:01.819663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.819625 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" containerID="cri-o://b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c" gracePeriod=45 Nov 04 03:11:01.826046 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.826010 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" containerID="cri-o://775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47" gracePeriod=45 Nov 04 03:11:01.860083 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.860048 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg"] Nov 04 03:11:01.860806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.860791 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afd3c41-3375-4ae7-8e07-e18676b0c3ec" containerName="istio-proxy" Nov 04 03:11:01.864364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.864350 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:01.924710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.924673 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:01.924904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.924853 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hgz\" (UniqueName: \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:01.924904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.924887 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:01.951525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.951492 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg"] Nov 04 03:11:01.964446 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:01.964377 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": dial tcp 10.133.0.109:8000: connect: connection refused" Nov 04 03:11:02.046280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.026412 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.046280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.026591 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hgz\" (UniqueName: \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.046280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.026626 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.046280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.028130 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.139549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.122322 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hgz\" (UniqueName: \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.139549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.122631 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1233a27-dq8tg\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.286427 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.286394 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 03:11:02.306459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.306429 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 03:11:02.308692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.308666 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 03:11:02.328328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.328234 4784 generic.go:356] "Generic (PLEG): container finished" podID="5db08a49-57b2-4e07-a805-be215c07b917" containerID="b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c" exitCode=0 Nov 04 03:11:02.328328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.328276 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" Nov 04 03:11:02.328328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.328316 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" event={"ID":"5db08a49-57b2-4e07-a805-be215c07b917","Type":"ContainerDied","Data":"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c"} Nov 04 03:11:02.328591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.328368 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x" event={"ID":"5db08a49-57b2-4e07-a805-be215c07b917","Type":"ContainerDied","Data":"d023b4dbe670df9d7153df46ff92fe5d765179bb504918398f1930ce79ae5179"} Nov 04 03:11:02.328591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.328393 4784 scope.go:122] "RemoveContainer" containerID="b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c" Nov 04 03:11:02.330199 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.330123 4784 generic.go:356] "Generic (PLEG): container finished" podID="f3727622-6296-404c-b05d-cf537d65ea4b" containerID="775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47" exitCode=0 Nov 04 03:11:02.330199 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.330163 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" Nov 04 03:11:02.330501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.330246 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" event={"ID":"f3727622-6296-404c-b05d-cf537d65ea4b","Type":"ContainerDied","Data":"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47"} Nov 04 03:11:02.330501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.330282 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl" event={"ID":"f3727622-6296-404c-b05d-cf537d65ea4b","Type":"ContainerDied","Data":"1af9875535b9dc5c52ee279a4b32219d7ee7d57fe72ba23ffda0cd5b5504ec48"} Nov 04 03:11:02.331759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.331736 4784 generic.go:356] "Generic (PLEG): container finished" podID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerID="c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09" exitCode=0 Nov 04 03:11:02.331874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.331796 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" Nov 04 03:11:02.331874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.331833 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" event={"ID":"e7fd5de9-0389-4968-876e-4ec6f0ebaf90","Type":"ContainerDied","Data":"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09"} Nov 04 03:11:02.331874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.331861 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf" event={"ID":"e7fd5de9-0389-4968-876e-4ec6f0ebaf90","Type":"ContainerDied","Data":"17289ad44f0751c12fcfdb0ccafa8758c195ff11a5f5b605c751f968b8e537e0"} Nov 04 03:11:02.344715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.342329 4784 scope.go:122] "RemoveContainer" containerID="b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c" Nov 04 03:11:02.344715 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:02.342699 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c\": container with ID starting with b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c not found: ID does not exist" containerID="b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c" Nov 04 03:11:02.344715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.342730 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c"} err="failed to get container status \"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c\": rpc error: code = NotFound desc = could not find container \"b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c\": container with ID starting with b97ed2f178679bb3dca2406b53326cde97fec339aefab8129fe872726d88c58c not found: ID does not exist" Nov 04 03:11:02.344715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.342747 4784 scope.go:122] "RemoveContainer" containerID="775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47" Nov 04 03:11:02.353786 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.353734 4784 scope.go:122] "RemoveContainer" containerID="775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47" Nov 04 03:11:02.354090 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:02.354056 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47\": container with ID starting with 775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47 not found: ID does not exist" containerID="775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47" Nov 04 03:11:02.354183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.354085 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47"} err="failed to get container status \"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47\": rpc error: code = NotFound desc = could not find container \"775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47\": container with ID starting with 775af7636139dd6d548a411be25a794d1196eef8ac9416e40a726334f8a5af47 not found: ID does not exist" Nov 04 03:11:02.354183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.354112 4784 scope.go:122] "RemoveContainer" containerID="c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09" Nov 04 03:11:02.372738 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.372712 4784 scope.go:122] "RemoveContainer" containerID="c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09" Nov 04 03:11:02.373555 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:02.373528 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09\": container with ID starting with c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09 not found: ID does not exist" containerID="c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09" Nov 04 03:11:02.373676 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.373570 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09"} err="failed to get container status \"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09\": rpc error: code = NotFound desc = could not find container \"c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09\": container with ID starting with c084b5965b71e315ae7c3cd2bbf6276ece71e764220f8702bfe7826659fdfb09 not found: ID does not exist" Nov 04 03:11:02.417821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.417533 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:02.438427 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438385 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.438621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438482 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.438621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438542 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.438621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438593 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438628 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438684 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438725 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\" (UniqueName: \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438771 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438815 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438846 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438884 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.438959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438918 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.438967 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.439027 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\") pod \"f3727622-6296-404c-b05d-cf537d65ea4b\" (UID: \"f3727622-6296-404c-b05d-cf537d65ea4b\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.439069 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\" (UniqueName: \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.439208 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\" (UniqueName: \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.439270 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\" (UniqueName: \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\") pod \"5db08a49-57b2-4e07-a805-be215c07b917\" (UID: \"5db08a49-57b2-4e07-a805-be215c07b917\") " Nov 04 03:11:02.439615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.439314 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\") pod \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\" (UID: \"e7fd5de9-0389-4968-876e-4ec6f0ebaf90\") " Nov 04 03:11:02.440137 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440109 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.440281 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440215 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.440373 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440350 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.440459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440438 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.440581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440559 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.440930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.440904 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.444700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.444639 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.444814 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.444700 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:02.444923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.444873 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.447275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.445358 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.461485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.461451 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "kube-api-access-lx8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:02.462069 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.462042 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm" pod "e7fd5de9-0389-4968-876e-4ec6f0ebaf90" (UID: "e7fd5de9-0389-4968-876e-4ec6f0ebaf90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.462174 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.462137 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.462276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.462240 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.462686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.462653 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "kube-api-access-ds5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:02.463963 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.463301 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "kube-api-access-tdz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:02.463963 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.463432 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs" pod "5db08a49-57b2-4e07-a805-be215c07b917" (UID: "5db08a49-57b2-4e07-a805-be215c07b917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:02.469673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.469629 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs" pod "f3727622-6296-404c-b05d-cf537d65ea4b" (UID: "f3727622-6296-404c-b05d-cf537d65ea4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:02.521428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.521392 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:11:02.543040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543000 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543046 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543061 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543073 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543087 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543100 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543116 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5rf\" (UniqueName: \"kubernetes.io/projected/f3727622-6296-404c-b05d-cf537d65ea4b-kube-api-access-ds5rf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543130 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db08a49-57b2-4e07-a805-be215c07b917-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543143 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543156 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543169 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543181 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543214 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3727622-6296-404c-b05d-cf537d65ea4b-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543226 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3727622-6296-404c-b05d-cf537d65ea4b-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.543238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543239 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db08a49-57b2-4e07-a805-be215c07b917-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.544011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543253 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8vz\" (UniqueName: \"kubernetes.io/projected/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-kube-api-access-lx8vz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.544011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543293 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdz45\" (UniqueName: \"kubernetes.io/projected/5db08a49-57b2-4e07-a805-be215c07b917-kube-api-access-tdz45\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.544011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.543309 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd5de9-0389-4968-876e-4ec6f0ebaf90-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.643835 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.643800 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\") pod \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " Nov 04 03:11:02.643943 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.643926 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\") pod \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " Nov 04 03:11:02.644071 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.644054 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\" (UniqueName: \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\") pod \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\" (UID: \"ad0129b8-9f5c-4278-a80f-d5068550ffa9\") " Nov 04 03:11:02.644366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.644335 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca" pod "ad0129b8-9f5c-4278-a80f-d5068550ffa9" (UID: "ad0129b8-9f5c-4278-a80f-d5068550ffa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:02.644587 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.644568 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.646085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.646064 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca" pod "ad0129b8-9f5c-4278-a80f-d5068550ffa9" (UID: "ad0129b8-9f5c-4278-a80f-d5068550ffa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:02.646379 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.646362 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z" pod "ad0129b8-9f5c-4278-a80f-d5068550ffa9" (UID: "ad0129b8-9f5c-4278-a80f-d5068550ffa9"). InnerVolumeSpecName "kube-api-access-2zf7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:02.755760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.755706 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 03:11:02.760743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.759793 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0129b8-9f5c-4278-a80f-d5068550ffa9-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.760743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.759829 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zf7z\" (UniqueName: \"kubernetes.io/projected/ad0129b8-9f5c-4278-a80f-d5068550ffa9-kube-api-access-2zf7z\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.829037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.829000 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 03:11:02.835791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.835764 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-w0-fe4e0c-kserve-d689c4b66-49rpf"] Nov 04 03:11:02.847270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.847237 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 03:11:02.881221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.881180 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg"] Nov 04 03:11:02.945844 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.945806 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 03:11:02.949432 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:02.949407 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/premium-w0-fe4e0c-kserve-dfd957b87-nsd5x"] Nov 04 03:11:03.216567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.216509 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db08a49-57b2-4e07-a805-be215c07b917" path="/var/lib/kubelet/pods/5db08a49-57b2-4e07-a805-be215c07b917/volumes" Nov 04 03:11:03.217495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.217470 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" path="/var/lib/kubelet/pods/e7fd5de9-0389-4968-876e-4ec6f0ebaf90/volumes" Nov 04 03:11:03.236308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.230292 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 03:11:03.236497 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.236473 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/simulator-w0-fe4e0c-kserve-54cddc7f46-8ztvl"] Nov 04 03:11:03.348453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.348405 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" event={"ID":"ad0129b8-9f5c-4278-a80f-d5068550ffa9","Type":"ContainerDied","Data":"2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10"} Nov 04 03:11:03.348881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.348467 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cec563528fd73cd08ac1e3c1f526e536504fb81e6b826fbc9db861c841f7f10" Nov 04 03:11:03.348881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.348538 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf" Nov 04 03:11:03.351474 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.351440 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" event={"ID":"6714fd46-a2b6-4b05-9eac-2ad039feef14","Type":"ContainerStarted","Data":"77186535e70530d37abe9f40a7feaeb873307c72d0ea94e72f7bc1b88a37be53"} Nov 04 03:11:03.652268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:03.652107 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" podStartSLOduration=2.6520860710000003 podStartE2EDuration="2.652086071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:03.649369268 +0000 UTC m=+2310.615870261" watchObservedRunningTime="2026-11-04 03:11:03.652086071 +0000 UTC m=+2310.618587064" Nov 04 03:11:04.142978 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.141885 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" containerID="cri-o://ad8d10bf39ddd85083df91455f829202d3bd81bbcd7f4d21e290df6de64b2885" gracePeriod=45 Nov 04 03:11:04.415285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.414155 4784 generic.go:356] "Generic (PLEG): container finished" podID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerID="ad8d10bf39ddd85083df91455f829202d3bd81bbcd7f4d21e290df6de64b2885" exitCode=0 Nov 04 03:11:04.415285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.414464 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" event={"ID":"2fa6c3fe-7c55-4688-b225-423d82cae782","Type":"ContainerDied","Data":"ad8d10bf39ddd85083df91455f829202d3bd81bbcd7f4d21e290df6de64b2885"} Nov 04 03:11:04.430212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.416801 4784 generic.go:356] "Generic (PLEG): container finished" podID="6714fd46-a2b6-4b05-9eac-2ad039feef14" containerID="1feb37c619adbe866b67923ab8f16a21550597ce66a5a1529e2f44527c3fccdf" exitCode=0 Nov 04 03:11:04.430212 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.416891 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" event={"ID":"6714fd46-a2b6-4b05-9eac-2ad039feef14","Type":"ContainerDied","Data":"1feb37c619adbe866b67923ab8f16a21550597ce66a5a1529e2f44527c3fccdf"} Nov 04 03:11:04.609214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:04.609027 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused" Nov 04 03:11:05.120221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.112087 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 03:11:05.150319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150164 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.150319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150234 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.150612 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150359 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.150612 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150461 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.150612 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150609 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.150801 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.150658 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\" (UniqueName: \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\") pod \"2fa6c3fe-7c55-4688-b225-423d82cae782\" (UID: \"2fa6c3fe-7c55-4688-b225-423d82cae782\") " Nov 04 03:11:05.151543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.151483 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.151707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.151580 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.151707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.151636 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.154559 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.154517 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.154670 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.154643 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "kube-api-access-vdjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:05.287216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.284356 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs" pod "2fa6c3fe-7c55-4688-b225-423d82cae782" (UID: "2fa6c3fe-7c55-4688-b225-423d82cae782"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.403661 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" podUID="9ad192d2-12d2-4da2-a496-a632d91889fc" containerName="payload-pre-processing" containerID="cri-o://788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50" gracePeriod=25 Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471329 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471364 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjmt\" (UniqueName: \"kubernetes.io/projected/2fa6c3fe-7c55-4688-b225-423d82cae782-kube-api-access-vdjmt\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471380 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471410 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471424 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa6c3fe-7c55-4688-b225-423d82cae782-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.488918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.471438 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa6c3fe-7c55-4688-b225-423d82cae782-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.589308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.588636 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" path="/var/lib/kubelet/pods/f3727622-6296-404c-b05d-cf537d65ea4b/volumes" Nov 04 03:11:05.599820 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.599789 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" Nov 04 03:11:05.602310 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.600048 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5" event={"ID":"2fa6c3fe-7c55-4688-b225-423d82cae782","Type":"ContainerDied","Data":"b7070a54d5fd7dd2b1c99f208199a9cbbe777426969913100b0a1896453e646e"} Nov 04 03:11:05.602310 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.601749 4784 scope.go:122] "RemoveContainer" containerID="ad8d10bf39ddd85083df91455f829202d3bd81bbcd7f4d21e290df6de64b2885" Nov 04 03:11:05.675983 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:05.675935 4784 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad192d2_12d2_4da2_a496_a632d91889fc.slice/crio-788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fa6c3fe_7c55_4688_b225_423d82cae782.slice/crio-b7070a54d5fd7dd2b1c99f208199a9cbbe777426969913100b0a1896453e646e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fa6c3fe_7c55_4688_b225_423d82cae782.slice\": RecentStats: unable to find data in memory cache]" Nov 04 03:11:05.792728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.792613 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 03:11:05.867349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.867320 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/premium-w1-e516ff-kserve-5684f77c78-vqlz5"] Nov 04 03:11:05.893364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.892980 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:05.915407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.915371 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 03:11:05.985584 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.985525 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:11:05.986244 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.986212 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" containerID="cri-o://a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa" gracePeriod=30 Nov 04 03:11:05.998488 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.998451 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\") pod \"6714fd46-a2b6-4b05-9eac-2ad039feef14\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " Nov 04 03:11:05.998662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.998541 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\") pod \"6714fd46-a2b6-4b05-9eac-2ad039feef14\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " Nov 04 03:11:05.998662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.998650 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\" (UniqueName: \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\") pod \"6714fd46-a2b6-4b05-9eac-2ad039feef14\" (UID: \"6714fd46-a2b6-4b05-9eac-2ad039feef14\") " Nov 04 03:11:05.998822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.998803 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\" (UniqueName: \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\") pod \"9ad192d2-12d2-4da2-a496-a632d91889fc\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " Nov 04 03:11:05.998880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.998842 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\") pod \"9ad192d2-12d2-4da2-a496-a632d91889fc\" (UID: \"9ad192d2-12d2-4da2-a496-a632d91889fc\") " Nov 04 03:11:06.000306 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.999733 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca" pod "6714fd46-a2b6-4b05-9eac-2ad039feef14" (UID: "6714fd46-a2b6-4b05-9eac-2ad039feef14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:06.000421 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:05.999925 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume" pod "9ad192d2-12d2-4da2-a496-a632d91889fc" (UID: "9ad192d2-12d2-4da2-a496-a632d91889fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:06.003017 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.002989 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca" pod "6714fd46-a2b6-4b05-9eac-2ad039feef14" (UID: "6714fd46-a2b6-4b05-9eac-2ad039feef14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:06.003951 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.003927 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz" pod "6714fd46-a2b6-4b05-9eac-2ad039feef14" (UID: "6714fd46-a2b6-4b05-9eac-2ad039feef14"). InnerVolumeSpecName "kube-api-access-k2hgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:06.004973 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.004937 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx" pod "9ad192d2-12d2-4da2-a496-a632d91889fc" (UID: "9ad192d2-12d2-4da2-a496-a632d91889fc"). InnerVolumeSpecName "kube-api-access-vb4qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:06.101075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.100978 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hgz\" (UniqueName: \"kubernetes.io/projected/6714fd46-a2b6-4b05-9eac-2ad039feef14-kube-api-access-k2hgz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.101075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.101017 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4qx\" (UniqueName: \"kubernetes.io/projected/9ad192d2-12d2-4da2-a496-a632d91889fc-kube-api-access-vb4qx\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.101075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.101033 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad192d2-12d2-4da2-a496-a632d91889fc-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.101075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.101050 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.101075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.101065 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6714fd46-a2b6-4b05-9eac-2ad039feef14-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.402257 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:06.402131 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:11:06.609378 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.609344 4784 generic.go:356] "Generic (PLEG): container finished" podID="9ad192d2-12d2-4da2-a496-a632d91889fc" containerID="788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50" exitCode=0 Nov 04 03:11:06.609866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.609401 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" Nov 04 03:11:06.609866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.609426 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" event={"ID":"9ad192d2-12d2-4da2-a496-a632d91889fc","Type":"ContainerDied","Data":"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50"} Nov 04 03:11:06.609866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.609482 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d" event={"ID":"9ad192d2-12d2-4da2-a496-a632d91889fc","Type":"ContainerDied","Data":"ad0d5e6353de747309642f2846eb02bd333396ceffa0e1a703b782b671874280"} Nov 04 03:11:06.609866 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.609505 4784 scope.go:122] "RemoveContainer" containerID="788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50" Nov 04 03:11:06.611033 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.611016 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" event={"ID":"6714fd46-a2b6-4b05-9eac-2ad039feef14","Type":"ContainerDied","Data":"77186535e70530d37abe9f40a7feaeb873307c72d0ea94e72f7bc1b88a37be53"} Nov 04 03:11:06.611111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.611044 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg" Nov 04 03:11:06.611154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.611049 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77186535e70530d37abe9f40a7feaeb873307c72d0ea94e72f7bc1b88a37be53" Nov 04 03:11:06.622171 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.622146 4784 scope.go:122] "RemoveContainer" containerID="788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50" Nov 04 03:11:06.622508 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:06.622490 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50\": container with ID starting with 788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50 not found: ID does not exist" containerID="788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50" Nov 04 03:11:06.622577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.622517 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50"} err="failed to get container status \"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50\": rpc error: code = NotFound desc = could not find container \"788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50\": container with ID starting with 788b024f817dd059470370960036a266b1519300dab6f9df5809bf66673baf50 not found: ID does not exist" Nov 04 03:11:06.779585 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.779540 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 03:11:06.783713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:06.783683 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe4e0c-8599b44f79-9xn4d"] Nov 04 03:11:07.161755 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.161676 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" path="/var/lib/kubelet/pods/2fa6c3fe-7c55-4688-b225-423d82cae782/volumes" Nov 04 03:11:07.162120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.162106 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad192d2-12d2-4da2-a496-a632d91889fc" path="/var/lib/kubelet/pods/9ad192d2-12d2-4da2-a496-a632d91889fc/volumes" Nov 04 03:11:07.259022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.258986 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd"] Nov 04 03:11:07.259645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259630 4784 cpu_manager.go:414] "containerMap: removing container" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259647 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259660 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259666 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259674 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6714fd46-a2b6-4b05-9eac-2ad039feef14" containerName="revoke-keys" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259680 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6714fd46-a2b6-4b05-9eac-2ad039feef14" containerName="revoke-keys" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259694 4784 cpu_manager.go:414] "containerMap: removing container" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" Nov 04 03:11:07.259701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259699 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259707 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259712 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259724 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9ad192d2-12d2-4da2-a496-a632d91889fc" containerName="payload-pre-processing" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259729 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad192d2-12d2-4da2-a496-a632d91889fc" containerName="payload-pre-processing" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259736 4784 cpu_manager.go:414] "containerMap: removing container" podUID="ad0129b8-9f5c-4278-a80f-d5068550ffa9" containerName="revoke-keys" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259752 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0129b8-9f5c-4278-a80f-d5068550ffa9" containerName="revoke-keys" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259854 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db08a49-57b2-4e07-a805-be215c07b917" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259864 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7fd5de9-0389-4968-876e-4ec6f0ebaf90" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259871 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6714fd46-a2b6-4b05-9eac-2ad039feef14" containerName="revoke-keys" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259877 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa6c3fe-7c55-4688-b225-423d82cae782" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259883 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3727622-6296-404c-b05d-cf537d65ea4b" containerName="main" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259890 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad192d2-12d2-4da2-a496-a632d91889fc" containerName="payload-pre-processing" Nov 04 03:11:07.259925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.259895 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0129b8-9f5c-4278-a80f-d5068550ffa9" containerName="revoke-keys" Nov 04 03:11:07.264802 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.264781 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.318806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.318772 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:11:07.318806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.318790 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:11:07.319075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.318824 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:11:07.319075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.318898 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:11:07.320528 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.320500 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd"] Nov 04 03:11:07.417633 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.417601 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.417812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.417658 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8gl\" (UniqueName: \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.417812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.417775 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.519599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.519555 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.520287 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.519633 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8gl\" (UniqueName: \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.520287 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.519713 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.520637 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.520617 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.525243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.523586 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.583625 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.583589 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8gl\" (UniqueName: \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\") pod \"maas-api-revoke-sub-57edbdda-gv9bd\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.756683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.756589 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" podUID="66543109-7ff8-4a52-b49a-cf9712812777" containerName="payload-processing" containerID="cri-o://e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b" gracePeriod=25 Nov 04 03:11:07.848631 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.848590 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" podUID="521cf223-72d7-45cf-b3a6-30871e1a5765" containerName="payload-pre-processing" containerID="cri-o://d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175" gracePeriod=25 Nov 04 03:11:07.878679 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.878648 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:07.975953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.975882 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:11:07.990913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:07.990876 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.062217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.055082 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-095081-openshift-default-dockercfg-pkjw8\"" Nov 04 03:11:08.137332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.137298 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:11:08.197673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197632 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.197880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197703 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78922\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.197880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197739 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.197880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197799 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.197880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197848 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.198231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197897 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.198231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197926 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.198231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.197977 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.198231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.198134 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.299902 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.299851 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.299948 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.299994 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78922\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300024 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300084 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300129 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300210 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300235 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.300483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.300285 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.301518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.301449 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.302791 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.302717 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.303747 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.303687 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.304849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.304794 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.304949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.304857 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd"] Nov 04 03:11:08.305430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.305382 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.308234 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.308165 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.308746 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.308685 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.347416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.347394 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 03:11:08.456965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.456933 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.457164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.457132 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78922\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\") pod \"e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.481331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.481303 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 03:11:08.505789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.505747 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\") pod \"66543109-7ff8-4a52-b49a-cf9712812777\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " Nov 04 03:11:08.506050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.506034 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\" (UniqueName: \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\") pod \"66543109-7ff8-4a52-b49a-cf9712812777\" (UID: \"66543109-7ff8-4a52-b49a-cf9712812777\") " Nov 04 03:11:08.506243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.506172 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume" pod "66543109-7ff8-4a52-b49a-cf9712812777" (UID: "66543109-7ff8-4a52-b49a-cf9712812777"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:08.506630 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.506608 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66543109-7ff8-4a52-b49a-cf9712812777-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:08.508425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.508400 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6" pod "66543109-7ff8-4a52-b49a-cf9712812777" (UID: "66543109-7ff8-4a52-b49a-cf9712812777"). InnerVolumeSpecName "kube-api-access-grdg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:08.607770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.607734 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\") pod \"521cf223-72d7-45cf-b3a6-30871e1a5765\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " Nov 04 03:11:08.607976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.607893 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\" (UniqueName: \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\") pod \"521cf223-72d7-45cf-b3a6-30871e1a5765\" (UID: \"521cf223-72d7-45cf-b3a6-30871e1a5765\") " Nov 04 03:11:08.608086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.608063 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume" pod "521cf223-72d7-45cf-b3a6-30871e1a5765" (UID: "521cf223-72d7-45cf-b3a6-30871e1a5765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:08.608408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.608385 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grdg6\" (UniqueName: \"kubernetes.io/projected/66543109-7ff8-4a52-b49a-cf9712812777-kube-api-access-grdg6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:08.608506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.608410 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf223-72d7-45cf-b3a6-30871e1a5765-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:08.610307 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.610280 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z" pod "521cf223-72d7-45cf-b3a6-30871e1a5765" (UID: "521cf223-72d7-45cf-b3a6-30871e1a5765"). InnerVolumeSpecName "kube-api-access-qzh2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:08.632198 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.632161 4784 generic.go:356] "Generic (PLEG): container finished" podID="521cf223-72d7-45cf-b3a6-30871e1a5765" containerID="d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175" exitCode=0 Nov 04 03:11:08.632374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.632222 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" Nov 04 03:11:08.632374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.632245 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" event={"ID":"521cf223-72d7-45cf-b3a6-30871e1a5765","Type":"ContainerDied","Data":"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175"} Nov 04 03:11:08.632374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.632300 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9" event={"ID":"521cf223-72d7-45cf-b3a6-30871e1a5765","Type":"ContainerDied","Data":"32a4e8ed2a433ed38e3b74c9ed96c58381350578dc6ae47f7adf1eb76a9afe17"} Nov 04 03:11:08.632374 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.632332 4784 scope.go:122] "RemoveContainer" containerID="d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175" Nov 04 03:11:08.634504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.634472 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" event={"ID":"16073ce3-a672-4271-909f-08f22bbb07a8","Type":"ContainerStarted","Data":"f4bbbb2f5a4854925b6fa8a13cc334514266ce3e578be284e36287116935fe1f"} Nov 04 03:11:08.636332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.636306 4784 generic.go:356] "Generic (PLEG): container finished" podID="66543109-7ff8-4a52-b49a-cf9712812777" containerID="e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b" exitCode=0 Nov 04 03:11:08.636445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.636374 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" event={"ID":"66543109-7ff8-4a52-b49a-cf9712812777","Type":"ContainerDied","Data":"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b"} Nov 04 03:11:08.636445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.636416 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" event={"ID":"66543109-7ff8-4a52-b49a-cf9712812777","Type":"ContainerDied","Data":"90c9c129799e411bce2ec3c6081b9ed13fe1cd1476e88b2831ddac42322c201a"} Nov 04 03:11:08.636445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.636376 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m" Nov 04 03:11:08.644665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.644643 4784 scope.go:122] "RemoveContainer" containerID="d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175" Nov 04 03:11:08.644967 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:08.644940 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175\": container with ID starting with d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175 not found: ID does not exist" containerID="d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175" Nov 04 03:11:08.645034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.644979 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175"} err="failed to get container status \"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175\": rpc error: code = NotFound desc = could not find container \"d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175\": container with ID starting with d4896d0ace289233253cfc9e7f665fdf2db6456aa51aff1c2639931417f78175 not found: ID does not exist" Nov 04 03:11:08.645034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.645000 4784 scope.go:122] "RemoveContainer" containerID="e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b" Nov 04 03:11:08.655596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.655571 4784 scope.go:122] "RemoveContainer" containerID="e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b" Nov 04 03:11:08.655951 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:08.655923 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b\": container with ID starting with e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b not found: ID does not exist" containerID="e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b" Nov 04 03:11:08.656021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.655965 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b"} err="failed to get container status \"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b\": rpc error: code = NotFound desc = could not find container \"e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b\": container with ID starting with e25627c1889de77eecfe8dd091f9e52a641874979f2000791ffff28f55ece68b not found: ID does not exist" Nov 04 03:11:08.709662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.709631 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzh2z\" (UniqueName: \"kubernetes.io/projected/521cf223-72d7-45cf-b3a6-30871e1a5765-kube-api-access-qzh2z\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:08.733680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.733642 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:08.807336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.807293 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 03:11:08.813599 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.813573 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e516ff-57b6c4bc94-5nc6m"] Nov 04 03:11:08.920412 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.920378 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 03:11:08.924204 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:08.924166 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e516ff-b7d8666c5-8jsr9"] Nov 04 03:11:09.126709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.126680 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:11:09.127176 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:11:09.127152 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b4b264_53bb_429c_8215_59856f253f65.slice/crio-c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd WatchSource:0}: Error finding container c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd: Status 404 returned error can't find the container with id c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd Nov 04 03:11:09.130644 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.130603 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:09.130737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.130700 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:09.130774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.130743 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:09.164068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.164040 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521cf223-72d7-45cf-b3a6-30871e1a5765" path="/var/lib/kubelet/pods/521cf223-72d7-45cf-b3a6-30871e1a5765/volumes" Nov 04 03:11:09.164591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.164573 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66543109-7ff8-4a52-b49a-cf9712812777" path="/var/lib/kubelet/pods/66543109-7ff8-4a52-b49a-cf9712812777/volumes" Nov 04 03:11:09.651840 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.651791 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerStarted","Data":"834cbb280dadc117d66c7e2158e4693953909d032a01f027100198c518c8c6ed"} Nov 04 03:11:09.652029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.651908 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerStarted","Data":"c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd"} Nov 04 03:11:09.656913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.656881 4784 generic.go:356] "Generic (PLEG): container finished" podID="16073ce3-a672-4271-909f-08f22bbb07a8" containerID="8c60c6770cc447563756bbe7ed9be155c729dad3dd8e85ae3a97bce2e59b7284" exitCode=0 Nov 04 03:11:09.657084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.656937 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" event={"ID":"16073ce3-a672-4271-909f-08f22bbb07a8","Type":"ContainerDied","Data":"8c60c6770cc447563756bbe7ed9be155c729dad3dd8e85ae3a97bce2e59b7284"} Nov 04 03:11:09.734122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.734078 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:09.779718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:09.779657 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podStartSLOduration=2.779639812 podStartE2EDuration="2.779639812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:09.774630685 +0000 UTC m=+2316.741131700" watchObservedRunningTime="2026-11-04 03:11:09.779639812 +0000 UTC m=+2316.746140803" Nov 04 03:11:10.342862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.342822 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq"] Nov 04 03:11:10.343718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343696 4784 cpu_manager.go:414] "containerMap: removing container" podUID="66543109-7ff8-4a52-b49a-cf9712812777" containerName="payload-processing" Nov 04 03:11:10.343774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343722 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66543109-7ff8-4a52-b49a-cf9712812777" containerName="payload-processing" Nov 04 03:11:10.343774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343767 4784 cpu_manager.go:414] "containerMap: removing container" podUID="521cf223-72d7-45cf-b3a6-30871e1a5765" containerName="payload-pre-processing" Nov 04 03:11:10.343847 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343776 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521cf223-72d7-45cf-b3a6-30871e1a5765" containerName="payload-pre-processing" Nov 04 03:11:10.343930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343919 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="66543109-7ff8-4a52-b49a-cf9712812777" containerName="payload-processing" Nov 04 03:11:10.343969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.343940 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="521cf223-72d7-45cf-b3a6-30871e1a5765" containerName="payload-pre-processing" Nov 04 03:11:10.347817 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.347797 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.404745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.404706 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq"] Nov 04 03:11:10.432073 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.432040 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.432269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.432235 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz86f\" (UniqueName: \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.432359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.432345 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.533686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.533640 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz86f\" (UniqueName: \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.533910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.533775 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.533910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.533850 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.534462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.534440 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.536649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.536618 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.598774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.598694 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz86f\" (UniqueName: \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\") pod \"maas-api-revoke-sub-a51b6610-f7kxq\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.665350 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.665256 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:10.742034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.741735 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:10.881928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.881899 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:10.940106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.940072 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\") pod \"16073ce3-a672-4271-909f-08f22bbb07a8\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " Nov 04 03:11:10.940636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.940160 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\" (UniqueName: \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\") pod \"16073ce3-a672-4271-909f-08f22bbb07a8\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " Nov 04 03:11:10.940636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.940285 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\") pod \"16073ce3-a672-4271-909f-08f22bbb07a8\" (UID: \"16073ce3-a672-4271-909f-08f22bbb07a8\") " Nov 04 03:11:10.940636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.940487 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca" pod "16073ce3-a672-4271-909f-08f22bbb07a8" (UID: "16073ce3-a672-4271-909f-08f22bbb07a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:10.940882 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.940798 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:10.942623 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.942599 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca" pod "16073ce3-a672-4271-909f-08f22bbb07a8" (UID: "16073ce3-a672-4271-909f-08f22bbb07a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:10.942737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:10.942636 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl" pod "16073ce3-a672-4271-909f-08f22bbb07a8" (UID: "16073ce3-a672-4271-909f-08f22bbb07a8"). InnerVolumeSpecName "kube-api-access-ml8gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:11.045597 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.045553 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8gl\" (UniqueName: \"kubernetes.io/projected/16073ce3-a672-4271-909f-08f22bbb07a8-kube-api-access-ml8gl\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.045597 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.045599 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16073ce3-a672-4271-909f-08f22bbb07a8-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.071582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.071487 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq"] Nov 04 03:11:11.076475 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:11:11.076442 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4629ecf_6050_436a_840b_3e10c0e659c6.slice/crio-0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf WatchSource:0}: Error finding container 0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf: Status 404 returned error can't find the container with id 0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf Nov 04 03:11:11.251864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.251838 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:11:11.348552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348511 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348567 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348644 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348679 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348738 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348832 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348844 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348863 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.348914 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349027 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:11.349068 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349052 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\") pod \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\" (UID: \"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a\") " Nov 04 03:11:11.349600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349168 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:11.349600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349394 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:11.349600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349425 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:11.349748 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349732 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.349841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349755 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.349841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349771 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.349841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349785 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.349841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.349798 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.351631 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.351613 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:11.351718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.351694 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "kube-api-access-z6jr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:11.351757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.351714 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:11.351757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.351734 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo" pod "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" (UID: "1dc57902-e9e8-4b48-ba1d-9a41e57fe03a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:11.451721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.451681 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.451721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.451712 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.451721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.451723 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jr2\" (UniqueName: \"kubernetes.io/projected/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-kube-api-access-z6jr2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.451969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.451733 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:11.689354 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.689309 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" event={"ID":"d4629ecf-6050-436a-840b-3e10c0e659c6","Type":"ContainerStarted","Data":"b2b7bab0ed68ebfcdbc1c1109dc7858c61332611cd5ce674d5e72ccbec85ca1d"} Nov 04 03:11:11.689589 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.689381 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" event={"ID":"d4629ecf-6050-436a-840b-3e10c0e659c6","Type":"ContainerStarted","Data":"0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf"} Nov 04 03:11:11.691344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.691316 4784 generic.go:356] "Generic (PLEG): container finished" podID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerID="a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa" exitCode=0 Nov 04 03:11:11.691484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.691369 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" Nov 04 03:11:11.691484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.691415 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerDied","Data":"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa"} Nov 04 03:11:11.691484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.691469 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk" event={"ID":"1dc57902-e9e8-4b48-ba1d-9a41e57fe03a","Type":"ContainerDied","Data":"1485598c5952e42d5ef709d5b54805372b263d138d5602436049cf89e3d314d7"} Nov 04 03:11:11.691583 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.691494 4784 scope.go:122] "RemoveContainer" containerID="a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa" Nov 04 03:11:11.696618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.693728 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" event={"ID":"16073ce3-a672-4271-909f-08f22bbb07a8","Type":"ContainerDied","Data":"f4bbbb2f5a4854925b6fa8a13cc334514266ce3e578be284e36287116935fe1f"} Nov 04 03:11:11.696618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.693770 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4bbbb2f5a4854925b6fa8a13cc334514266ce3e578be284e36287116935fe1f" Nov 04 03:11:11.696618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.693767 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd" Nov 04 03:11:11.696618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.693958 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:11.696618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.695035 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:11:11.706535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.706515 4784 scope.go:122] "RemoveContainer" containerID="ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c" Nov 04 03:11:11.717994 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.717971 4784 scope.go:122] "RemoveContainer" containerID="a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa" Nov 04 03:11:11.719008 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:11.718952 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa\": container with ID starting with a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa not found: ID does not exist" containerID="a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa" Nov 04 03:11:11.719180 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.719013 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa"} err="failed to get container status \"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa\": rpc error: code = NotFound desc = could not find container \"a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa\": container with ID starting with a7755ff45f85f9555e949dd4c2d781e388939af6fca2f5611fae88aa31eb4aaa not found: ID does not exist" Nov 04 03:11:11.719180 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.719052 4784 scope.go:122] "RemoveContainer" containerID="ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c" Nov 04 03:11:11.729750 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:11.729714 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c\": container with ID starting with ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c not found: ID does not exist" containerID="ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c" Nov 04 03:11:11.729946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.729756 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c"} err="failed to get container status \"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c\": rpc error: code = NotFound desc = could not find container \"ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c\": container with ID starting with ed22a074cc7d68a8803113c5f220ac2e7a43d5bac71a37b2a26acdeb59e19b9c not found: ID does not exist" Nov 04 03:11:11.813086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.813021 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" podStartSLOduration=1.813007415 podStartE2EDuration="1.813007415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:11.810942321 +0000 UTC m=+2318.777443313" watchObservedRunningTime="2026-11-04 03:11:11.813007415 +0000 UTC m=+2318.779508406" Nov 04 03:11:11.981012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.980976 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:11:11.987931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:11.987899 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9d2b96-openshift-default-cc56c4dff-6w9vk"] Nov 04 03:11:12.706179 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:12.705983 4784 generic.go:356] "Generic (PLEG): container finished" podID="d4629ecf-6050-436a-840b-3e10c0e659c6" containerID="b2b7bab0ed68ebfcdbc1c1109dc7858c61332611cd5ce674d5e72ccbec85ca1d" exitCode=0 Nov 04 03:11:12.706179 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:12.706085 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" event={"ID":"d4629ecf-6050-436a-840b-3e10c0e659c6","Type":"ContainerDied","Data":"b2b7bab0ed68ebfcdbc1c1109dc7858c61332611cd5ce674d5e72ccbec85ca1d"} Nov 04 03:11:13.161675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.161582 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" path="/var/lib/kubelet/pods/1dc57902-e9e8-4b48-ba1d-9a41e57fe03a/volumes" Nov 04 03:11:13.883621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.883591 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:13.990655 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.990561 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\" (UniqueName: \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\") pod \"d4629ecf-6050-436a-840b-3e10c0e659c6\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " Nov 04 03:11:13.990655 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.990625 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\") pod \"d4629ecf-6050-436a-840b-3e10c0e659c6\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " Nov 04 03:11:13.990861 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.990746 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\") pod \"d4629ecf-6050-436a-840b-3e10c0e659c6\" (UID: \"d4629ecf-6050-436a-840b-3e10c0e659c6\") " Nov 04 03:11:13.991035 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.991008 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca" pod "d4629ecf-6050-436a-840b-3e10c0e659c6" (UID: "d4629ecf-6050-436a-840b-3e10c0e659c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:13.992997 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.992973 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f" pod "d4629ecf-6050-436a-840b-3e10c0e659c6" (UID: "d4629ecf-6050-436a-840b-3e10c0e659c6"). InnerVolumeSpecName "kube-api-access-lz86f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:13.993129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:13.993111 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca" pod "d4629ecf-6050-436a-840b-3e10c0e659c6" (UID: "d4629ecf-6050-436a-840b-3e10c0e659c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:14.091854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.091808 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz86f\" (UniqueName: \"kubernetes.io/projected/d4629ecf-6050-436a-840b-3e10c0e659c6-kube-api-access-lz86f\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:14.091854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.091845 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:14.091854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.091862 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4629ecf-6050-436a-840b-3e10c0e659c6-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:14.584255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.584222 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 03:11:14.747693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.747662 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" Nov 04 03:11:14.747854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.747662 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq" event={"ID":"d4629ecf-6050-436a-840b-3e10c0e659c6","Type":"ContainerDied","Data":"0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf"} Nov 04 03:11:14.747854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:14.747814 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f180eeff08d5abe1c5cb6f6bca668fcb7383764bb13a9d7b5ec3097fa9fdebf" Nov 04 03:11:19.059249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059208 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5"] Nov 04 03:11:19.059853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059838 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059855 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059869 4784 cpu_manager.go:414] "containerMap: removing container" podUID="16073ce3-a672-4271-909f-08f22bbb07a8" containerName="revoke-keys" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059874 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16073ce3-a672-4271-909f-08f22bbb07a8" containerName="revoke-keys" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059888 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059893 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.059903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059902 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d4629ecf-6050-436a-840b-3e10c0e659c6" containerName="revoke-keys" Nov 04 03:11:19.060095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.059907 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4629ecf-6050-436a-840b-3e10c0e659c6" containerName="revoke-keys" Nov 04 03:11:19.060095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.060018 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.060095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.060026 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="16073ce3-a672-4271-909f-08f22bbb07a8" containerName="revoke-keys" Nov 04 03:11:19.060095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.060034 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc57902-e9e8-4b48-ba1d-9a41e57fe03a" containerName="istio-proxy" Nov 04 03:11:19.060095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.060040 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4629ecf-6050-436a-840b-3e10c0e659c6" containerName="revoke-keys" Nov 04 03:11:19.070019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.069972 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.120286 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.120249 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5"] Nov 04 03:11:19.166840 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.166795 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:11:19.166840 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.166836 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:11:19.167061 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.166855 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:11:19.167061 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.166836 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:11:19.175674 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.175645 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.175949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.175924 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676gp\" (UniqueName: \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.176035 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.175973 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.276884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.276842 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676gp\" (UniqueName: \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.277075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.276892 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.277075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.276928 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.277704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.277682 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.279651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.279597 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.339812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.339731 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676gp\" (UniqueName: \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\") pod \"maas-api-revoke-sub-1f53f08d-dc5r5\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.387744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.387707 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:19.771397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.771360 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5"] Nov 04 03:11:19.771778 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:11:19.771753 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0534750f_a52f_4dcb_a291_747edbb2ac5c.slice/crio-402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c WatchSource:0}: Error finding container 402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c: Status 404 returned error can't find the container with id 402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c Nov 04 03:11:19.791987 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:19.791950 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" event={"ID":"0534750f-a52f-4dcb-a291-747edbb2ac5c","Type":"ContainerStarted","Data":"402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c"} Nov 04 03:11:20.822216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:20.812569 4784 generic.go:356] "Generic (PLEG): container finished" podID="0534750f-a52f-4dcb-a291-747edbb2ac5c" containerID="5642e393584bf12901e2072bbbe4131e677c6558731d400092e41a1d505f832c" exitCode=0 Nov 04 03:11:20.822216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:20.812847 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" event={"ID":"0534750f-a52f-4dcb-a291-747edbb2ac5c","Type":"ContainerDied","Data":"5642e393584bf12901e2072bbbe4131e677c6558731d400092e41a1d505f832c"} Nov 04 03:11:22.061804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.061774 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:22.152508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.152413 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\") pod \"0534750f-a52f-4dcb-a291-747edbb2ac5c\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " Nov 04 03:11:22.152508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.152464 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\" (UniqueName: \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\") pod \"0534750f-a52f-4dcb-a291-747edbb2ac5c\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " Nov 04 03:11:22.152760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.152686 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\") pod \"0534750f-a52f-4dcb-a291-747edbb2ac5c\" (UID: \"0534750f-a52f-4dcb-a291-747edbb2ac5c\") " Nov 04 03:11:22.152877 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.152803 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca" pod "0534750f-a52f-4dcb-a291-747edbb2ac5c" (UID: "0534750f-a52f-4dcb-a291-747edbb2ac5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:22.153288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.153202 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:22.154918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.154889 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp" pod "0534750f-a52f-4dcb-a291-747edbb2ac5c" (UID: "0534750f-a52f-4dcb-a291-747edbb2ac5c"). InnerVolumeSpecName "kube-api-access-676gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:22.154918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.154904 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca" pod "0534750f-a52f-4dcb-a291-747edbb2ac5c" (UID: "0534750f-a52f-4dcb-a291-747edbb2ac5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:22.255021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.254861 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0534750f-a52f-4dcb-a291-747edbb2ac5c-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:22.255021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.254901 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676gp\" (UniqueName: \"kubernetes.io/projected/0534750f-a52f-4dcb-a291-747edbb2ac5c-kube-api-access-676gp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:22.865661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.865628 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" Nov 04 03:11:22.865884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.865680 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5" event={"ID":"0534750f-a52f-4dcb-a291-747edbb2ac5c","Type":"ContainerDied","Data":"402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c"} Nov 04 03:11:22.865884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:22.865761 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402baedd54eadb0671483168f5231a0b8e1732957bb0ae1e0fe86813ecedcb6c" Nov 04 03:11:23.341910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.341874 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm"] Nov 04 03:11:23.342812 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.342796 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0534750f-a52f-4dcb-a291-747edbb2ac5c" containerName="revoke-keys" Nov 04 03:11:23.342864 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.342815 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0534750f-a52f-4dcb-a291-747edbb2ac5c" containerName="revoke-keys" Nov 04 03:11:23.342957 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.342946 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0534750f-a52f-4dcb-a291-747edbb2ac5c" containerName="revoke-keys" Nov 04 03:11:23.346928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.346903 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.401747 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.401699 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:11:23.411704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.411655 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm"] Nov 04 03:11:23.452860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.452830 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:11:23.463558 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.463526 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:11:23.463764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.463593 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:11:23.475307 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.475269 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnkg\" (UniqueName: \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.475488 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.475312 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.475488 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.475335 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.577331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.577289 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnkg\" (UniqueName: \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.577331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.577337 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.577573 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.577359 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.578203 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.578165 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.580027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.579999 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.650266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.646567 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnkg\" (UniqueName: \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\") pod \"maas-api-revoke-sub-22f10719-wkrrm\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:23.660774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:23.660744 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:24.129549 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:11:24.129519 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7dbec3_f271_4601_b2a8_15ba25d6b744.slice/crio-269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508 WatchSource:0}: Error finding container 269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508: Status 404 returned error can't find the container with id 269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508 Nov 04 03:11:24.130548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:24.130515 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm"] Nov 04 03:11:24.902934 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:24.902805 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" event={"ID":"aa7dbec3-f271-4601-b2a8-15ba25d6b744","Type":"ContainerStarted","Data":"a0d888cf956dd9d279d41838ce3ed02c8bbb423a197b44d6e605de85430d8589"} Nov 04 03:11:24.902934 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:24.902911 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" event={"ID":"aa7dbec3-f271-4601-b2a8-15ba25d6b744","Type":"ContainerStarted","Data":"269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508"} Nov 04 03:11:25.022151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:25.022099 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" podStartSLOduration=2.022084092 podStartE2EDuration="2.022084092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:25.019973379 +0000 UTC m=+2331.986474374" watchObservedRunningTime="2026-11-04 03:11:25.022084092 +0000 UTC m=+2331.988585083" Nov 04 03:11:25.912613 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:25.912575 4784 generic.go:356] "Generic (PLEG): container finished" podID="aa7dbec3-f271-4601-b2a8-15ba25d6b744" containerID="a0d888cf956dd9d279d41838ce3ed02c8bbb423a197b44d6e605de85430d8589" exitCode=0 Nov 04 03:11:25.913101 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:25.912661 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" event={"ID":"aa7dbec3-f271-4601-b2a8-15ba25d6b744","Type":"ContainerDied","Data":"a0d888cf956dd9d279d41838ce3ed02c8bbb423a197b44d6e605de85430d8589"} Nov 04 03:11:27.303749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.303302 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:27.362400 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.362361 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\") pod \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " Nov 04 03:11:27.362768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.362744 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\" (UniqueName: \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\") pod \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " Nov 04 03:11:27.363308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.363285 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\") pod \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\" (UID: \"aa7dbec3-f271-4601-b2a8-15ba25d6b744\") " Nov 04 03:11:27.364129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.362869 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca" pod "aa7dbec3-f271-4601-b2a8-15ba25d6b744" (UID: "aa7dbec3-f271-4601-b2a8-15ba25d6b744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:27.364129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.364097 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.367057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.367031 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg" pod "aa7dbec3-f271-4601-b2a8-15ba25d6b744" (UID: "aa7dbec3-f271-4601-b2a8-15ba25d6b744"). InnerVolumeSpecName "kube-api-access-mqnkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:27.372710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.372664 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca" pod "aa7dbec3-f271-4601-b2a8-15ba25d6b744" (UID: "aa7dbec3-f271-4601-b2a8-15ba25d6b744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:27.467445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.467406 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnkg\" (UniqueName: \"kubernetes.io/projected/aa7dbec3-f271-4601-b2a8-15ba25d6b744-kube-api-access-mqnkg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.467725 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.467704 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7dbec3-f271-4601-b2a8-15ba25d6b744-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.958777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.958744 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" Nov 04 03:11:27.959022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.958877 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm" event={"ID":"aa7dbec3-f271-4601-b2a8-15ba25d6b744","Type":"ContainerDied","Data":"269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508"} Nov 04 03:11:27.959022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:27.958922 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="269da108fcb735e1e11568513baa155dac56e7461132dfcc46d26381317e3508" Nov 04 03:11:29.606203 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.606149 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" containerID="cri-o://9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad" gracePeriod=45 Nov 04 03:11:29.692841 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.692799 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": dial tcp 10.133.0.124:8000: connect: connection refused" Nov 04 03:11:29.937079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.937045 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 03:11:29.975636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.975601 4784 generic.go:356] "Generic (PLEG): container finished" podID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerID="9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad" exitCode=0 Nov 04 03:11:29.975636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.975646 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" Nov 04 03:11:29.975890 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.975683 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" event={"ID":"28572dfe-e4cc-4127-b9b5-ad50dbd85367","Type":"ContainerDied","Data":"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad"} Nov 04 03:11:29.975890 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.975727 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r" event={"ID":"28572dfe-e4cc-4127-b9b5-ad50dbd85367","Type":"ContainerDied","Data":"3125ae7df3db0e41da56d5827452d76040b4e0cc65d77a8d7fc451879e7405fd"} Nov 04 03:11:29.975890 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:29.975744 4784 scope.go:122] "RemoveContainer" containerID="9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad" Nov 04 03:11:30.039085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.039042 4784 scope.go:122] "RemoveContainer" containerID="9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad" Nov 04 03:11:30.039611 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:30.039573 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad\": container with ID starting with 9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad not found: ID does not exist" containerID="9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad" Nov 04 03:11:30.039744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.039624 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad"} err="failed to get container status \"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad\": rpc error: code = NotFound desc = could not find container \"9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad\": container with ID starting with 9a1fd6edd69287bdf93ac9c4b9774f4ed59160c857daabba2cfda80323c3e5ad not found: ID does not exist" Nov 04 03:11:30.048673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.048635 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.048865 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.048727 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.048865 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.048776 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.048996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.048945 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\" (UniqueName: \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.048996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.048977 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\" (UniqueName: \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.049123 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049051 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:30.049123 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049080 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\") pod \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\" (UID: \"28572dfe-e4cc-4127-b9b5-ad50dbd85367\") " Nov 04 03:11:30.049263 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049232 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:30.052242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049699 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:30.052242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049722 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.052242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.049746 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.058460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.058408 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:30.058460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.058436 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "kube-api-access-p4jsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:30.059359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.059327 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs" pod "28572dfe-e4cc-4127-b9b5-ad50dbd85367" (UID: "28572dfe-e4cc-4127-b9b5-ad50dbd85367"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:30.150503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.150466 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.150503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.150502 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jsg\" (UniqueName: \"kubernetes.io/projected/28572dfe-e4cc-4127-b9b5-ad50dbd85367-kube-api-access-p4jsg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.150722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.150516 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.150722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.150528 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28572dfe-e4cc-4127-b9b5-ad50dbd85367-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.401208 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.401154 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 03:11:30.408532 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:30.408496 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-2-w3-cdcf67-kserve-cb8c7b996-xzj8r"] Nov 04 03:11:31.161703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:31.161654 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" path="/var/lib/kubelet/pods/28572dfe-e4cc-4127-b9b5-ad50dbd85367/volumes" Nov 04 03:11:31.771955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:31.771917 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 03:11:31.772270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:31.772242 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" containerID="cri-o://06b157aa881f72e9ee55f0c012315e0537546b8cd87f066ee1ac6a66f57d2666" gracePeriod=30 Nov 04 03:11:33.247546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:33.247513 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 03:11:33.248154 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:33.247744 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" containerID="cri-o://7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6" gracePeriod=30 Nov 04 03:11:34.419616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:34.419572 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 03:11:36.403040 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:36.402994 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:11:37.045008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.044936 4784 generic.go:356] "Generic (PLEG): container finished" podID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerID="06b157aa881f72e9ee55f0c012315e0537546b8cd87f066ee1ac6a66f57d2666" exitCode=0 Nov 04 03:11:37.045172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.045004 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerDied","Data":"06b157aa881f72e9ee55f0c012315e0537546b8cd87f066ee1ac6a66f57d2666"} Nov 04 03:11:37.045172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.045059 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" event={"ID":"96afbb02-6497-4a26-9975-f0340b4e0bd1","Type":"ContainerDied","Data":"2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c"} Nov 04 03:11:37.045172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.045082 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1386419617dd202658b3dee51ad47da75baca144752aa078402a8c0301a04c" Nov 04 03:11:37.045172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.045105 4784 scope.go:122] "RemoveContainer" containerID="9c81c4ed1d9b44d88fcb04b53736f93080a1e71c3893e9f2138737ea3ff28730" Nov 04 03:11:37.049135 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.049111 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 03:11:37.099163 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099115 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099268 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099312 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099366 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099411 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099407 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.099640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099598 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099750 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099653 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099750 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099726 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099780 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\") pod \"96afbb02-6497-4a26-9975-f0340b4e0bd1\" (UID: \"96afbb02-6497-4a26-9975-f0340b4e0bd1\") " Nov 04 03:11:37.099946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.099885 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.100031 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100009 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.100232 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100147 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.100341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100073 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:37.100408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100360 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.100408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100382 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.100408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100397 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.100518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.100413 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.101923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.101895 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:37.102054 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.101943 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "kube-api-access-5w5b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:37.102169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.102153 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:37.102244 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.102201 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy" pod "96afbb02-6497-4a26-9975-f0340b4e0bd1" (UID: "96afbb02-6497-4a26-9975-f0340b4e0bd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.201749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.201715 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.201749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.201747 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5b2\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-kube-api-access-5w5b2\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.201749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.201757 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.202007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.201766 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96afbb02-6497-4a26-9975-f0340b4e0bd1-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.202007 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:37.201777 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96afbb02-6497-4a26-9975-f0340b4e0bd1-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.053892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.053861 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6" Nov 04 03:11:38.347224 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.347119 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 03:11:38.351240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.351214 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-tzgt6"] Nov 04 03:11:38.518393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.518364 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 03:11:38.619680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.619586 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.619680 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.619645 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.619900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.619685 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.619900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.619809 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620010 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.619991 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.620061 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620017 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620102 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620065 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:38.620102 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620076 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620178 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620128 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620273 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620216 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620273 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620248 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\") pod \"a4816a13-78c7-4e20-b914-4c589fadfc16\" (UID: \"a4816a13-78c7-4e20-b914-4c589fadfc16\") " Nov 04 03:11:38.620404 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620305 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.620520 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620497 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.620606 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.620520 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.621112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.621088 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.621266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.621116 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.621266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.621132 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4816a13-78c7-4e20-b914-4c589fadfc16-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.621266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.621156 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.621266 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.621172 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.622417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.622392 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:38.622535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.622480 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:38.622708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.622683 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.622822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.622725 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk" pod "a4816a13-78c7-4e20-b914-4c589fadfc16" (UID: "a4816a13-78c7-4e20-b914-4c589fadfc16"). InnerVolumeSpecName "kube-api-access-z6cwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:38.722500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.722463 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6cwk\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-kube-api-access-z6cwk\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.722500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.722504 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.722771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.722529 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.722771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:38.722543 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4816a13-78c7-4e20-b914-4c589fadfc16-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:39.063037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.062997 4784 generic.go:356] "Generic (PLEG): container finished" podID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerID="7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6" exitCode=0 Nov 04 03:11:39.063617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.063057 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" Nov 04 03:11:39.063617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.063063 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" event={"ID":"a4816a13-78c7-4e20-b914-4c589fadfc16","Type":"ContainerDied","Data":"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6"} Nov 04 03:11:39.063617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.063119 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws" event={"ID":"a4816a13-78c7-4e20-b914-4c589fadfc16","Type":"ContainerDied","Data":"59f9620d38b3a9c4d2c733943cc6522c94ada4aa6e83fdcf5f218a616262162b"} Nov 04 03:11:39.063617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.063146 4784 scope.go:122] "RemoveContainer" containerID="7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6" Nov 04 03:11:39.075080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.075056 4784 scope.go:122] "RemoveContainer" containerID="7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6" Nov 04 03:11:39.075550 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:11:39.075525 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6\": container with ID starting with 7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6 not found: ID does not exist" containerID="7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6" Nov 04 03:11:39.075645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.075558 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6"} err="failed to get container status \"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6\": rpc error: code = NotFound desc = could not find container \"7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6\": container with ID starting with 7e958c7abf101466a4ed7676f96560ef07f732a16f2f2ab906009dedc160a1f6 not found: ID does not exist" Nov 04 03:11:39.160331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.160297 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" path="/var/lib/kubelet/pods/96afbb02-6497-4a26-9975-f0340b4e0bd1/volumes" Nov 04 03:11:39.235106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.235072 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 03:11:39.286167 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.286130 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-9qgws"] Nov 04 03:11:39.420806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:39.420744 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" containerID="cri-o://06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281" gracePeriod=25 Nov 04 03:11:41.161102 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:41.161067 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" path="/var/lib/kubelet/pods/a4816a13-78c7-4e20-b914-4c589fadfc16/volumes" Nov 04 03:11:43.597291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597255 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:11:43.597854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597841 4784 cpu_manager.go:414] "containerMap: removing container" podUID="aa7dbec3-f271-4601-b2a8-15ba25d6b744" containerName="revoke-keys" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597855 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7dbec3-f271-4601-b2a8-15ba25d6b744" containerName="revoke-keys" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597871 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597876 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597889 4784 cpu_manager.go:414] "containerMap: removing container" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597894 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:11:43.597919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597920 4784 cpu_manager.go:414] "containerMap: removing container" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597925 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597934 4784 cpu_manager.go:414] "containerMap: removing container" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.597938 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.598028 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7dbec3-f271-4601-b2a8-15ba25d6b744" containerName="revoke-keys" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.598038 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4816a13-78c7-4e20-b914-4c589fadfc16" containerName="istio-proxy" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.598046 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:11:43.598112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.598056 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="28572dfe-e4cc-4127-b9b5-ad50dbd85367" containerName="main" Nov 04 03:11:43.603092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.603066 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.657868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.657836 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ebb7fd-openshift-default-dockercfg-k9smx\"" Nov 04 03:11:43.684356 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684321 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975wz\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684542 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684447 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684542 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684484 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684542 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684516 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684544 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684696 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684846 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684829 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684914 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684869 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.684973 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.684913 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.716423 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.716386 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:11:43.786426 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786388 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786426 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786431 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786471 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786519 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975wz\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786607 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786640 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786674 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.786713 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786711 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786779 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786865 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.786934 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787137 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.787061 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.787150 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.787504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.787479 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.789132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.789106 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.789360 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.789339 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.848919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.848839 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975wz\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.848919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.848842 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:43.916792 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:43.916750 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:44.351983 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:11:44.351945 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278fad12_4775_4580_82de_d5fe0fa3e9b1.slice/crio-702af01211d498ad14c6dd2c07f6ccf60cf68df10d4590fd68c1e76a444aeca0 WatchSource:0}: Error finding container 702af01211d498ad14c6dd2c07f6ccf60cf68df10d4590fd68c1e76a444aeca0: Status 404 returned error can't find the container with id 702af01211d498ad14c6dd2c07f6ccf60cf68df10d4590fd68c1e76a444aeca0 Nov 04 03:11:44.352819 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:44.352768 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:11:44.354673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:44.354640 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:44.354756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:44.354707 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:44.354756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:44.354735 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:11:45.097820 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.097786 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:11:45.110053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.110018 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" event={"ID":"278fad12-4775-4580-82de-d5fe0fa3e9b1","Type":"ContainerStarted","Data":"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b"} Nov 04 03:11:45.110297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.110084 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" event={"ID":"278fad12-4775-4580-82de-d5fe0fa3e9b1","Type":"ContainerStarted","Data":"702af01211d498ad14c6dd2c07f6ccf60cf68df10d4590fd68c1e76a444aeca0"} Nov 04 03:11:45.287941 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.287886 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podStartSLOduration=2.287869824 podStartE2EDuration="2.287869824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:45.280349814 +0000 UTC m=+2352.246850807" watchObservedRunningTime="2026-11-04 03:11:45.287869824 +0000 UTC m=+2352.254370816" Nov 04 03:11:45.917223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.917172 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:45.919247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.919223 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:45.919367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:45.919275 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:46.918105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:46.918067 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:46.918705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:46.918131 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:47.917944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:47.917907 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:47.918144 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:47.917968 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:48.917790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:48.917747 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:48.917993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:48.917808 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:49.918163 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:49.918116 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:49.918664 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:49.918239 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:50.917581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:50.917536 4784 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 04 03:11:50.917782 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:50.917610 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 04 03:11:51.922750 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:51.922715 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:51.923478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:51.923455 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:51.925268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:51.924388 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:52.830917 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:52.830864 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:11:53.169803 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:53.169770 4784 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" secret="" err="secret \"e2e-worker-w6-ebb7fd-openshift-default-dockercfg-k9smx\" not found" Nov 04 03:11:54.176868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:54.176820 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" containerID="cri-o://7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b" gracePeriod=30 Nov 04 03:11:55.098285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:55.098247 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:11:59.555515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.555486 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617033 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617097 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617136 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617180 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617305 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617399 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617430 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617557 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.617643 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\") pod \"278fad12-4775-4580-82de-d5fe0fa3e9b1\" (UID: \"278fad12-4775-4580-82de-d5fe0fa3e9b1\") " Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.618667 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.618857 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.620374 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.620569 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:59.641903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.620816 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:59.654498 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.654445 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:59.655736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.655699 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "kube-api-access-975wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:59.656503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.656474 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:59.657820 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.657792 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo" pod "278fad12-4775-4580-82de-d5fe0fa3e9b1" (UID: "278fad12-4775-4580-82de-d5fe0fa3e9b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:59.706145 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.706078 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 03:11:59.719040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.718998 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719038 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719055 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719066 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719075 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719083 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719092 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975wz\" (UniqueName: \"kubernetes.io/projected/278fad12-4775-4580-82de-d5fe0fa3e9b1-kube-api-access-975wz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719101 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278fad12-4775-4580-82de-d5fe0fa3e9b1-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.719290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.719109 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278fad12-4775-4580-82de-d5fe0fa3e9b1-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.819937 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.819902 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\" (UniqueName: \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\") pod \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " Nov 04 03:11:59.820132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.819946 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\") pod \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\" (UID: \"4ea2964d-1f82-40ca-855f-211bf8b60f0e\") " Nov 04 03:11:59.820412 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.820390 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume" pod "4ea2964d-1f82-40ca-855f-211bf8b60f0e" (UID: "4ea2964d-1f82-40ca-855f-211bf8b60f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:59.820557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.820539 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea2964d-1f82-40ca-855f-211bf8b60f0e-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.822317 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.822299 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf" pod "4ea2964d-1f82-40ca-855f-211bf8b60f0e" (UID: "4ea2964d-1f82-40ca-855f-211bf8b60f0e"). InnerVolumeSpecName "kube-api-access-qvbhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:59.921881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:11:59.921843 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbhf\" (UniqueName: \"kubernetes.io/projected/4ea2964d-1f82-40ca-855f-211bf8b60f0e-kube-api-access-qvbhf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:00.220845 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.220810 4784 generic.go:356] "Generic (PLEG): container finished" podID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerID="06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281" exitCode=0 Nov 04 03:12:00.221019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.220855 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" Nov 04 03:12:00.221019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.220889 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" event={"ID":"4ea2964d-1f82-40ca-855f-211bf8b60f0e","Type":"ContainerDied","Data":"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281"} Nov 04 03:12:00.221019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.220942 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm" event={"ID":"4ea2964d-1f82-40ca-855f-211bf8b60f0e","Type":"ContainerDied","Data":"ca95f639f251bc6257df8255d63ca9abfee95a99cd3ff11ff4a991aa74c58c3c"} Nov 04 03:12:00.221019 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.220965 4784 scope.go:122] "RemoveContainer" containerID="06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281" Nov 04 03:12:00.222523 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.222501 4784 generic.go:356] "Generic (PLEG): container finished" podID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerID="7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b" exitCode=0 Nov 04 03:12:00.222665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.222548 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" Nov 04 03:12:00.222665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.222589 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" event={"ID":"278fad12-4775-4580-82de-d5fe0fa3e9b1","Type":"ContainerDied","Data":"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b"} Nov 04 03:12:00.222665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.222633 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z" event={"ID":"278fad12-4775-4580-82de-d5fe0fa3e9b1","Type":"ContainerDied","Data":"702af01211d498ad14c6dd2c07f6ccf60cf68df10d4590fd68c1e76a444aeca0"} Nov 04 03:12:00.237414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.237389 4784 scope.go:122] "RemoveContainer" containerID="06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281" Nov 04 03:12:00.237766 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:12:00.237744 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281\": container with ID starting with 06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281 not found: ID does not exist" containerID="06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281" Nov 04 03:12:00.237842 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.237774 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281"} err="failed to get container status \"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281\": rpc error: code = NotFound desc = could not find container \"06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281\": container with ID starting with 06dd2ddbf36cdd40ac8239fe79a6944604046adb1c8208523266eb4531701281 not found: ID does not exist" Nov 04 03:12:00.237842 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.237795 4784 scope.go:122] "RemoveContainer" containerID="7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b" Nov 04 03:12:00.248385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.248362 4784 scope.go:122] "RemoveContainer" containerID="7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b" Nov 04 03:12:00.248644 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:12:00.248624 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b\": container with ID starting with 7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b not found: ID does not exist" containerID="7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b" Nov 04 03:12:00.248687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.248655 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b"} err="failed to get container status \"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b\": rpc error: code = NotFound desc = could not find container \"7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b\": container with ID starting with 7ba0b9a654d13238561396129c7281432aa8fafcf4b25f791de0ac6052aee21b not found: ID does not exist" Nov 04 03:12:00.392661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.392623 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:12:00.442306 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.442264 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-r2z9z"] Nov 04 03:12:00.534789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.534684 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 03:12:00.539079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:00.539048 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cdcf67-dc57c4455-ljnxm"] Nov 04 03:12:01.161493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:01.161461 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" path="/var/lib/kubelet/pods/278fad12-4775-4580-82de-d5fe0fa3e9b1/volumes" Nov 04 03:12:01.161906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:01.161891 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" path="/var/lib/kubelet/pods/4ea2964d-1f82-40ca-855f-211bf8b60f0e/volumes" Nov 04 03:12:06.402342 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:12:06.402294 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:12:15.519552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:15.519467 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 03:12:15.520080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:15.519684 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" containerID="cri-o://be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3" gracePeriod=30 Nov 04 03:12:20.800399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.800335 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 03:12:20.817434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817408 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817475 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817509 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817530 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817557 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817611 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817658 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817685 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817753 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\") pod \"da87486b-e9f0-4206-9f05-37d685d811bd\" (UID: \"da87486b-e9f0-4206-9f05-37d685d811bd\") " Nov 04 03:12:20.817986 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817854 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.818049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.817984 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.818110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818046 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.818247 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818227 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.818327 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818252 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:12:20.818327 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818303 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.818327 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818319 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.818430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818328 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.818430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.818338 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.819947 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.819927 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "kube-api-access-vkdrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:12:20.820366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.820338 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:12:20.820366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.820353 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:12:20.820366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.820359 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy" pod "da87486b-e9f0-4206-9f05-37d685d811bd" (UID: "da87486b-e9f0-4206-9f05-37d685d811bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.919693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.919657 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.919693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.919691 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87486b-e9f0-4206-9f05-37d685d811bd-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.919693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.919700 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87486b-e9f0-4206-9f05-37d685d811bd-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.919693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.919709 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87486b-e9f0-4206-9f05-37d685d811bd-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.920063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:20.919727 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdrp\" (UniqueName: \"kubernetes.io/projected/da87486b-e9f0-4206-9f05-37d685d811bd-kube-api-access-vkdrp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:21.416176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.416138 4784 generic.go:356] "Generic (PLEG): container finished" podID="da87486b-e9f0-4206-9f05-37d685d811bd" containerID="be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3" exitCode=0 Nov 04 03:12:21.416391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.416226 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" event={"ID":"da87486b-e9f0-4206-9f05-37d685d811bd","Type":"ContainerDied","Data":"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3"} Nov 04 03:12:21.416391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.416289 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" event={"ID":"da87486b-e9f0-4206-9f05-37d685d811bd","Type":"ContainerDied","Data":"5cda08020cf4c738dbf65febe2ef678421401f5b7a3a72b5bc3d39062e6e1db6"} Nov 04 03:12:21.416391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.416312 4784 scope.go:122] "RemoveContainer" containerID="be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3" Nov 04 03:12:21.416391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.416238 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf" Nov 04 03:12:21.428353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.428332 4784 scope.go:122] "RemoveContainer" containerID="be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3" Nov 04 03:12:21.428642 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:12:21.428624 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3\": container with ID starting with be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3 not found: ID does not exist" containerID="be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3" Nov 04 03:12:21.428693 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.428651 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3"} err="failed to get container status \"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3\": rpc error: code = NotFound desc = could not find container \"be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3\": container with ID starting with be4e4f4d0f8921ca82d3e023711689a02adb1caf4828283843b9481c149a0fd3 not found: ID does not exist" Nov 04 03:12:21.587363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.587330 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 03:12:21.637826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:21.637795 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-w5ldf"] Nov 04 03:12:23.160813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:23.160768 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" path="/var/lib/kubelet/pods/da87486b-e9f0-4206-9f05-37d685d811bd/volumes" Nov 04 03:12:33.790766 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:33.790734 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:12:33.801725 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:33.801696 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:12:33.802860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:33.802829 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:12:33.807854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:33.807827 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:12:36.389330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:36.389294 4784 scope.go:122] "RemoveContainer" containerID="06b157aa881f72e9ee55f0c012315e0537546b8cd87f066ee1ac6a66f57d2666" Nov 04 03:12:36.406039 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:12:36.405997 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:12:50.693954 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.693921 4784 generic.go:356] "Generic (PLEG): container finished" podID="f7b4b264-53bb-429c-8215-59856f253f65" containerID="834cbb280dadc117d66c7e2158e4693953909d032a01f027100198c518c8c6ed" exitCode=0 Nov 04 03:12:50.694435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.693995 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerDied","Data":"834cbb280dadc117d66c7e2158e4693953909d032a01f027100198c518c8c6ed"} Nov 04 03:12:50.694435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.694419 4784 scope.go:122] "RemoveContainer" containerID="834cbb280dadc117d66c7e2158e4693953909d032a01f027100198c518c8c6ed" Nov 04 03:12:50.695859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.695824 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:12:50.695964 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.695932 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:12:50.696014 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:50.695973 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:12:51.703262 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:51.703211 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerStarted","Data":"dc955c4c77f306b444048e24b93fc11802c38893fb727abeb460368d26409ad9"} Nov 04 03:12:51.734581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:51.734540 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:12:51.734776 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:51.734754 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:51.734822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:51.734802 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:52.734900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:52.734866 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:52.735369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:52.734919 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:53.734453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:53.734417 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:53.734677 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:53.734473 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:54.734984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:54.734951 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:54.735458 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:54.735009 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:55.734122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:55.734092 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:55.734393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:55.734368 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:56.694774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:56.694735 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:12:56.734229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:56.733962 4784 patch_prober.go:29] interesting pod/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 04 03:12:56.734229 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:56.734014 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 04 03:12:57.738952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:57.738917 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:12:57.740119 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:12:57.740093 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:13:06.400872 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:13:06.400827 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:13:36.401437 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:13:36.401392 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:13:47.461530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.461437 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl"] Nov 04 03:13:47.462291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462269 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" Nov 04 03:13:47.462407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462293 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" Nov 04 03:13:47.462407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462331 4784 cpu_manager.go:414] "containerMap: removing container" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" Nov 04 03:13:47.462407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462339 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" Nov 04 03:13:47.462407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462380 4784 cpu_manager.go:414] "containerMap: removing container" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" Nov 04 03:13:47.462407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462388 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" Nov 04 03:13:47.462671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462529 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="96afbb02-6497-4a26-9975-f0340b4e0bd1" containerName="istio-proxy" Nov 04 03:13:47.462671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462544 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea2964d-1f82-40ca-855f-211bf8b60f0e" containerName="payload-pre-processing" Nov 04 03:13:47.462671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462559 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="da87486b-e9f0-4206-9f05-37d685d811bd" containerName="istio-proxy" Nov 04 03:13:47.462671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.462573 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="278fad12-4775-4580-82de-d5fe0fa3e9b1" containerName="istio-proxy" Nov 04 03:13:47.466166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.466143 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.522370 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.522335 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl"] Nov 04 03:13:47.569168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.569129 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:13:47.569410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.569352 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:13:47.569410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.569365 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:13:47.570801 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.570784 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:13:47.620645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.620604 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.620868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.620649 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj6k\" (UniqueName: \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.620868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.620684 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.722049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.721956 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.722049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.721998 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj6k\" (UniqueName: \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.722049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.722022 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.722893 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.722822 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.724785 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.724763 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:47.784971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:47.784938 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj6k\" (UniqueName: \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\") pod \"maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:48.080549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:48.080466 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:48.460856 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:13:48.460821 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c6e3f3_5a4d_47b3_9906_d9a14f1c0bd1.slice/crio-8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad WatchSource:0}: Error finding container 8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad: Status 404 returned error can't find the container with id 8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad Nov 04 03:13:48.461795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:48.461727 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl"] Nov 04 03:13:49.137827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:49.137737 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" event={"ID":"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1","Type":"ContainerStarted","Data":"d8991027ed5bbecbb712de8e67d6c153d2bfeab6d38f6c8727d27138bdd07d3b"} Nov 04 03:13:49.137827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:49.137796 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" event={"ID":"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1","Type":"ContainerStarted","Data":"8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad"} Nov 04 03:13:49.256823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:49.256770 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" podStartSLOduration=2.25675654 podStartE2EDuration="2.25675654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:13:49.255579407 +0000 UTC m=+2476.222080400" watchObservedRunningTime="2026-11-04 03:13:49.25675654 +0000 UTC m=+2476.223257532" Nov 04 03:13:50.146867 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:50.146835 4784 generic.go:356] "Generic (PLEG): container finished" podID="06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" containerID="d8991027ed5bbecbb712de8e67d6c153d2bfeab6d38f6c8727d27138bdd07d3b" exitCode=0 Nov 04 03:13:50.147410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:50.146894 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" event={"ID":"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1","Type":"ContainerDied","Data":"d8991027ed5bbecbb712de8e67d6c153d2bfeab6d38f6c8727d27138bdd07d3b"} Nov 04 03:13:51.318215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.318170 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:51.360249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.360216 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\") pod \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " Nov 04 03:13:51.360451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.360277 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\") pod \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " Nov 04 03:13:51.360451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.360316 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\" (UniqueName: \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\") pod \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\" (UID: \"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1\") " Nov 04 03:13:51.360683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.360655 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca" pod "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" (UID: "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:13:51.362686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.362662 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca" pod "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" (UID: "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:13:51.362769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.362678 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k" pod "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" (UID: "06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1"). InnerVolumeSpecName "kube-api-access-bwj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:13:51.462211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.462163 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:13:51.462434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.462224 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:13:51.462434 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:51.462235 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwj6k\" (UniqueName: \"kubernetes.io/projected/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1-kube-api-access-bwj6k\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:13:52.162430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:52.162398 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" Nov 04 03:13:52.162608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:52.162431 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl" event={"ID":"06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1","Type":"ContainerDied","Data":"8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad"} Nov 04 03:13:52.162608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:52.162490 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8698eb4f50ad484d95376b8666b7f2f175710bf1a9c8badf6836525defe5f5ad" Nov 04 03:13:58.987579 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:13:58.987542 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:14:03.271729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.271685 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92"] Nov 04 03:14:03.272269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.272254 4784 cpu_manager.go:414] "containerMap: removing container" podUID="06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" containerName="revoke-keys" Nov 04 03:14:03.272269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.272269 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" containerName="revoke-keys" Nov 04 03:14:03.272396 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.272386 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" containerName="revoke-keys" Nov 04 03:14:03.275926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.275907 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.334277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.334238 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.334480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.334316 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92"] Nov 04 03:14:03.334480 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.334430 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zxc\" (UniqueName: \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.334609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.334515 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.379427 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.379366 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:14:03.379427 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.379417 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:14:03.379674 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.379454 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:14:03.380349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.380330 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:14:03.436078 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.436041 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.436314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.436135 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zxc\" (UniqueName: \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.436314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.436180 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.436698 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.436676 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.438671 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.438650 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.498834 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.498790 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zxc\" (UniqueName: \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\") pod \"maas-api-revoke-sub-57c8ceee-rzq92\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.590476 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.590372 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:03.973317 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.973284 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92"] Nov 04 03:14:03.973444 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:14:03.973429 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fecbb7_500a_4adc_b321_a0234f9d21f1.slice/crio-15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e WatchSource:0}: Error finding container 15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e: Status 404 returned error can't find the container with id 15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e Nov 04 03:14:03.987985 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:03.987951 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" podUID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" containerName="payload-processing" containerID="cri-o://15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd" gracePeriod=25 Nov 04 03:14:04.246329 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.246297 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:14:04.252089 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.252061 4784 generic.go:356] "Generic (PLEG): container finished" podID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" containerID="15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd" exitCode=0 Nov 04 03:14:04.252254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.252108 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" Nov 04 03:14:04.252254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.252144 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" event={"ID":"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0","Type":"ContainerDied","Data":"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd"} Nov 04 03:14:04.252339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.252249 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw" event={"ID":"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0","Type":"ContainerDied","Data":"85f960803480eeb2060de917fe3b37ef413f1436065daf6d33ae1d7818a36a10"} Nov 04 03:14:04.252339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.252284 4784 scope.go:122] "RemoveContainer" containerID="15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd" Nov 04 03:14:04.253505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.253473 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" event={"ID":"71fecbb7-500a-4adc-b321-a0234f9d21f1","Type":"ContainerStarted","Data":"15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e"} Nov 04 03:14:04.264385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.264357 4784 scope.go:122] "RemoveContainer" containerID="15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd" Nov 04 03:14:04.264673 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:14:04.264653 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd\": container with ID starting with 15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd not found: ID does not exist" containerID="15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd" Nov 04 03:14:04.264733 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.264681 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd"} err="failed to get container status \"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd\": rpc error: code = NotFound desc = could not find container \"15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd\": container with ID starting with 15cf7e6d91e01490921304091339c3842191f1eb2bbbeba44f851a67632081bd not found: ID does not exist" Nov 04 03:14:04.345734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.345699 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\") pod \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " Nov 04 03:14:04.346142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.345798 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\" (UniqueName: \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\") pod \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\" (UID: \"45de264f-b1d0-4425-9b7f-dd4a8d0df5c0\") " Nov 04 03:14:04.346142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.346068 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume" pod "45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" (UID: "45de264f-b1d0-4425-9b7f-dd4a8d0df5c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:14:04.346279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.346152 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:04.348045 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.348023 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9" pod "45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" (UID: "45de264f-b1d0-4425-9b7f-dd4a8d0df5c0"). InnerVolumeSpecName "kube-api-access-zxvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:14:04.447471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.447431 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvk9\" (UniqueName: \"kubernetes.io/projected/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0-kube-api-access-zxvk9\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:04.677940 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.677906 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:14:04.681419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:04.681392 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-8vnkw"] Nov 04 03:14:05.161722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:05.161624 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" path="/var/lib/kubelet/pods/45de264f-b1d0-4425-9b7f-dd4a8d0df5c0/volumes" Nov 04 03:14:05.263686 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:05.263595 4784 generic.go:356] "Generic (PLEG): container finished" podID="71fecbb7-500a-4adc-b321-a0234f9d21f1" containerID="6186537a435cfb3c8356784a8de09882b181c0f3ddc43d58e3bded9882ff5757" exitCode=0 Nov 04 03:14:05.263845 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:05.263680 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" event={"ID":"71fecbb7-500a-4adc-b321-a0234f9d21f1","Type":"ContainerDied","Data":"6186537a435cfb3c8356784a8de09882b181c0f3ddc43d58e3bded9882ff5757"} Nov 04 03:14:06.402251 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:14:06.402208 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:14:06.437390 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.437364 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:06.569718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.569637 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\") pod \"71fecbb7-500a-4adc-b321-a0234f9d21f1\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " Nov 04 03:14:06.569862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.569720 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\") pod \"71fecbb7-500a-4adc-b321-a0234f9d21f1\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " Nov 04 03:14:06.569862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.569815 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\" (UniqueName: \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\") pod \"71fecbb7-500a-4adc-b321-a0234f9d21f1\" (UID: \"71fecbb7-500a-4adc-b321-a0234f9d21f1\") " Nov 04 03:14:06.570133 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.570105 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca" pod "71fecbb7-500a-4adc-b321-a0234f9d21f1" (UID: "71fecbb7-500a-4adc-b321-a0234f9d21f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:14:06.570340 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.570324 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:06.572695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.572662 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca" pod "71fecbb7-500a-4adc-b321-a0234f9d21f1" (UID: "71fecbb7-500a-4adc-b321-a0234f9d21f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:14:06.572799 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.572702 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc" pod "71fecbb7-500a-4adc-b321-a0234f9d21f1" (UID: "71fecbb7-500a-4adc-b321-a0234f9d21f1"). InnerVolumeSpecName "kube-api-access-86zxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:14:06.671450 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.671411 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fecbb7-500a-4adc-b321-a0234f9d21f1-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:06.671450 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:06.671444 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zxc\" (UniqueName: \"kubernetes.io/projected/71fecbb7-500a-4adc-b321-a0234f9d21f1-kube-api-access-86zxc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:07.280553 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:07.280515 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" Nov 04 03:14:07.280553 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:07.280540 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92" event={"ID":"71fecbb7-500a-4adc-b321-a0234f9d21f1","Type":"ContainerDied","Data":"15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e"} Nov 04 03:14:07.280743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:07.280590 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15663844b031dae08b387eef5dee55008985cb4783c2026839c3454e80e51c8e" Nov 04 03:14:09.061397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:09.061363 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv"] Nov 04 03:14:09.112617 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:09.112580 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44249e5b-prgbv"] Nov 04 03:14:09.160913 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:09.160882 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e8a204-95ba-41a4-8a4f-142432a04b91" path="/var/lib/kubelet/pods/f5e8a204-95ba-41a4-8a4f-142432a04b91/volumes" Nov 04 03:14:22.050231 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:22.050169 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl"] Nov 04 03:14:22.107047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:22.107007 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb708d2-p9hxl"] Nov 04 03:14:23.161166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:23.161132 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4799f0b-61e8-4b28-b9b5-29f80077a213" path="/var/lib/kubelet/pods/a4799f0b-61e8-4b28-b9b5-29f80077a213/volumes" Nov 04 03:14:36.400710 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:14:36.400673 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:14:36.449720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:36.449683 4784 scope.go:122] "RemoveContainer" containerID="82eb1b3c6d139e7efa713ea34f72b1cf6277e043ed018bc6ad3315b288257140" Nov 04 03:14:44.795543 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:44.795507 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl"] Nov 04 03:14:44.846367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:44.846336 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cbcf738-s8vbl"] Nov 04 03:14:45.161074 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:14:45.160995 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1" path="/var/lib/kubelet/pods/06c6e3f3-5a4d-47b3-9906-d9a14f1c0bd1/volumes" Nov 04 03:15:00.178123 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.178082 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:00.179697 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179664 4784 cpu_manager.go:414] "containerMap: removing container" podUID="71fecbb7-500a-4adc-b321-a0234f9d21f1" containerName="revoke-keys" Nov 04 03:15:00.179697 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179698 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fecbb7-500a-4adc-b321-a0234f9d21f1" containerName="revoke-keys" Nov 04 03:15:00.179879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179745 4784 cpu_manager.go:414] "containerMap: removing container" podUID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" containerName="payload-processing" Nov 04 03:15:00.179879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179754 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" containerName="payload-processing" Nov 04 03:15:00.179949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179933 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fecbb7-500a-4adc-b321-a0234f9d21f1" containerName="revoke-keys" Nov 04 03:15:00.179984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.179957 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="45de264f-b1d0-4425-9b7f-dd4a8d0df5c0" containerName="payload-processing" Nov 04 03:15:00.184183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.184162 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.184467 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.184438 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:00.189146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.189127 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.189275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.189226 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:00.192953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.192934 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.198147 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.198122 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:00.201901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.201883 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.215810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.215706 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.215810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.215750 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2v26\" (UniqueName: \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.238096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.238064 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:15:00.238096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.238083 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:15:00.238381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.238087 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:15:00.317862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.317797 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2v26\" (UniqueName: \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.317883 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxd7\" (UniqueName: \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.317948 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.317995 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.318036 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfxx\" (UniqueName: \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.318127 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.318182 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcctn\" (UniqueName: \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.318376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.318256 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.319116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.319078 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.320282 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.320259 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:00.325972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.325894 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:00.331052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.331030 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:00.382676 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.382612 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2v26\" (UniqueName: \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.384207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.384164 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:00.419523 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419486 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419549 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419593 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfxx\" (UniqueName: \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419683 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419749 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcctn\" (UniqueName: \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.419876 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxd7\" (UniqueName: \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.420486 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.420999 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.421210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.421025 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.482780 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.482703 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcctn\" (UniqueName: \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\") pod \"maas-api-key-cleanup-29896035-wkzf6\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.482928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.482808 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxd7\" (UniqueName: \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.482928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.482808 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfxx\" (UniqueName: \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\") pod \"maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.499568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.499541 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:00.510271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.507742 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:00.515403 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.515373 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:00.521704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.521681 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:00.909276 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:15:00.909237 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524920e9_b313_4888_9fe3_c292c98b405a.slice/crio-f8eb633dee21767ef6999dec53ba35db743b0f6bbb52d2d924cad56eeae64195 WatchSource:0}: Error finding container f8eb633dee21767ef6999dec53ba35db743b0f6bbb52d2d924cad56eeae64195: Status 404 returned error can't find the container with id f8eb633dee21767ef6999dec53ba35db743b0f6bbb52d2d924cad56eeae64195 Nov 04 03:15:00.909883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.909763 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:00.959481 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:15:00.959449 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec13397_ce3a_4b29_a74a_b2e15ecba84d.slice/crio-0874f2f3b2b7f0215b3497b8955aa8a6466be05aa5ab425111f97dae0584fd10 WatchSource:0}: Error finding container 0874f2f3b2b7f0215b3497b8955aa8a6466be05aa5ab425111f97dae0584fd10: Status 404 returned error can't find the container with id 0874f2f3b2b7f0215b3497b8955aa8a6466be05aa5ab425111f97dae0584fd10 Nov 04 03:15:00.959605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.959499 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:00.959938 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:15:00.959916 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28f67c9_86d4_43db_90fb_40252a0da44d.slice/crio-75dbfdff492b218554ae45017af6a9a7cf6b3a468aa9830069b602bcc71910ee WatchSource:0}: Error finding container 75dbfdff492b218554ae45017af6a9a7cf6b3a468aa9830069b602bcc71910ee: Status 404 returned error can't find the container with id 75dbfdff492b218554ae45017af6a9a7cf6b3a468aa9830069b602bcc71910ee Nov 04 03:15:00.962269 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:15:00.962246 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f9282_abcb_487e_af52_e756e925714a.slice/crio-ed7049383c6b06fed7238a506b2350f8e1561539082b8ba396a723fadb4feca7 WatchSource:0}: Error finding container ed7049383c6b06fed7238a506b2350f8e1561539082b8ba396a723fadb4feca7: Status 404 returned error can't find the container with id ed7049383c6b06fed7238a506b2350f8e1561539082b8ba396a723fadb4feca7 Nov 04 03:15:00.964233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.964207 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:00.968148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:00.968108 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:01.691949 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.691907 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerStarted","Data":"65cf4d89236ffe0a153326212f06a34fcd986a90657a336c7702bd373d3bce5a"} Nov 04 03:15:01.692509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.691970 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerStarted","Data":"f8eb633dee21767ef6999dec53ba35db743b0f6bbb52d2d924cad56eeae64195"} Nov 04 03:15:01.693777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.693745 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerStarted","Data":"c06e28cf83faacbba94fe889acf45335d1aa00a39bc81339d58a28c488ad6ea1"} Nov 04 03:15:01.693944 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.693792 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerStarted","Data":"ed7049383c6b06fed7238a506b2350f8e1561539082b8ba396a723fadb4feca7"} Nov 04 03:15:01.695493 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.695465 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerStarted","Data":"6a9f21964dca78f0c96f1a96957789f1bb6cf21be92304af9cc5ecc05a570b1d"} Nov 04 03:15:01.695644 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.695509 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerStarted","Data":"75dbfdff492b218554ae45017af6a9a7cf6b3a468aa9830069b602bcc71910ee"} Nov 04 03:15:01.702141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.702113 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerStarted","Data":"05b1e293bf15417ce31209e075f97ff1a79d7b86e108ee60c86036209562371f"} Nov 04 03:15:01.702277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.702172 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerStarted","Data":"0874f2f3b2b7f0215b3497b8955aa8a6466be05aa5ab425111f97dae0584fd10"} Nov 04 03:15:01.812771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.812716 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" podStartSLOduration=1.81269784 podStartE2EDuration="1.81269784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:15:01.809254807 +0000 UTC m=+2548.775755840" watchObservedRunningTime="2026-11-04 03:15:01.81269784 +0000 UTC m=+2548.779198833" Nov 04 03:15:01.927970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:01.927914 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" podStartSLOduration=1.927901866 podStartE2EDuration="1.927901866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:15:01.927098989 +0000 UTC m=+2548.893599973" watchObservedRunningTime="2026-11-04 03:15:01.927901866 +0000 UTC m=+2548.894402857" Nov 04 03:15:02.047999 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.047946 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" podStartSLOduration=2.047932605 podStartE2EDuration="2.047932605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:15:02.044930384 +0000 UTC m=+2549.011431377" watchObservedRunningTime="2026-11-04 03:15:02.047932605 +0000 UTC m=+2549.014433597" Nov 04 03:15:02.711176 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.711143 4784 generic.go:356] "Generic (PLEG): container finished" podID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerID="05b1e293bf15417ce31209e075f97ff1a79d7b86e108ee60c86036209562371f" exitCode=60 Nov 04 03:15:02.711688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.711227 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerDied","Data":"05b1e293bf15417ce31209e075f97ff1a79d7b86e108ee60c86036209562371f"} Nov 04 03:15:02.711688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.711499 4784 scope.go:122] "RemoveContainer" containerID="05b1e293bf15417ce31209e075f97ff1a79d7b86e108ee60c86036209562371f" Nov 04 03:15:02.713282 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.713258 4784 generic.go:356] "Generic (PLEG): container finished" podID="524920e9-b313-4888-9fe3-c292c98b405a" containerID="65cf4d89236ffe0a153326212f06a34fcd986a90657a336c7702bd373d3bce5a" exitCode=60 Nov 04 03:15:02.713365 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.713344 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerDied","Data":"65cf4d89236ffe0a153326212f06a34fcd986a90657a336c7702bd373d3bce5a"} Nov 04 03:15:02.713570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.713553 4784 scope.go:122] "RemoveContainer" containerID="65cf4d89236ffe0a153326212f06a34fcd986a90657a336c7702bd373d3bce5a" Nov 04 03:15:02.715099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.715074 4784 generic.go:356] "Generic (PLEG): container finished" podID="e13f9282-abcb-487e-af52-e756e925714a" containerID="c06e28cf83faacbba94fe889acf45335d1aa00a39bc81339d58a28c488ad6ea1" exitCode=60 Nov 04 03:15:02.715209 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.715175 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerDied","Data":"c06e28cf83faacbba94fe889acf45335d1aa00a39bc81339d58a28c488ad6ea1"} Nov 04 03:15:02.715418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.715400 4784 scope.go:122] "RemoveContainer" containerID="c06e28cf83faacbba94fe889acf45335d1aa00a39bc81339d58a28c488ad6ea1" Nov 04 03:15:02.717341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.717316 4784 generic.go:356] "Generic (PLEG): container finished" podID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerID="6a9f21964dca78f0c96f1a96957789f1bb6cf21be92304af9cc5ecc05a570b1d" exitCode=60 Nov 04 03:15:02.717445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.717403 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerDied","Data":"6a9f21964dca78f0c96f1a96957789f1bb6cf21be92304af9cc5ecc05a570b1d"} Nov 04 03:15:02.717600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.717585 4784 scope.go:122] "RemoveContainer" containerID="6a9f21964dca78f0c96f1a96957789f1bb6cf21be92304af9cc5ecc05a570b1d" Nov 04 03:15:02.832886 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:02.832827 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" podStartSLOduration=2.83281196 podStartE2EDuration="2.83281196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:15:02.162568067 +0000 UTC m=+2549.129069059" watchObservedRunningTime="2026-11-04 03:15:02.83281196 +0000 UTC m=+2549.799312951" Nov 04 03:15:03.726774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.726729 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerStarted","Data":"bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9"} Nov 04 03:15:03.728772 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.728744 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerStarted","Data":"d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6"} Nov 04 03:15:03.730581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.730551 4784 generic.go:356] "Generic (PLEG): container finished" podID="e13f9282-abcb-487e-af52-e756e925714a" containerID="d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9" exitCode=60 Nov 04 03:15:03.730721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.730630 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerDied","Data":"d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9"} Nov 04 03:15:03.730721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.730685 4784 scope.go:122] "RemoveContainer" containerID="c06e28cf83faacbba94fe889acf45335d1aa00a39bc81339d58a28c488ad6ea1" Nov 04 03:15:03.730844 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.730822 4784 scope.go:122] "RemoveContainer" containerID="d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9" Nov 04 03:15:03.731138 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:03.731116 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb_odh-ai-gateway-infra(e13f9282-abcb-487e-af52-e756e925714a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" podUID="e13f9282-abcb-487e-af52-e756e925714a" Nov 04 03:15:03.733978 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:03.733954 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerStarted","Data":"2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7"} Nov 04 03:15:04.743770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.743739 4784 generic.go:356] "Generic (PLEG): container finished" podID="524920e9-b313-4888-9fe3-c292c98b405a" containerID="d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6" exitCode=60 Nov 04 03:15:04.744301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.743827 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerDied","Data":"d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6"} Nov 04 03:15:04.744301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.743901 4784 scope.go:122] "RemoveContainer" containerID="65cf4d89236ffe0a153326212f06a34fcd986a90657a336c7702bd373d3bce5a" Nov 04 03:15:04.744301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.744133 4784 scope.go:122] "RemoveContainer" containerID="d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6" Nov 04 03:15:04.744506 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:04.744487 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps_odh-ai-gateway-infra(524920e9-b313-4888-9fe3-c292c98b405a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" podUID="524920e9-b313-4888-9fe3-c292c98b405a" Nov 04 03:15:04.746012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.745993 4784 scope.go:122] "RemoveContainer" containerID="d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9" Nov 04 03:15:04.746293 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:04.746273 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb_odh-ai-gateway-infra(e13f9282-abcb-487e-af52-e756e925714a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" podUID="e13f9282-abcb-487e-af52-e756e925714a" Nov 04 03:15:04.747709 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.747689 4784 generic.go:356] "Generic (PLEG): container finished" podID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerID="2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7" exitCode=60 Nov 04 03:15:04.747828 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.747769 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerDied","Data":"2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7"} Nov 04 03:15:04.748006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.747989 4784 scope.go:122] "RemoveContainer" containerID="2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7" Nov 04 03:15:04.748311 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:04.748285 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt_odh-ai-gateway-infra(c28f67c9-86d4-43db-90fb-40252a0da44d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" Nov 04 03:15:04.749764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.749745 4784 generic.go:356] "Generic (PLEG): container finished" podID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerID="bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9" exitCode=60 Nov 04 03:15:04.749854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.749825 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerDied","Data":"bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9"} Nov 04 03:15:04.750025 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.750009 4784 scope.go:122] "RemoveContainer" containerID="bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9" Nov 04 03:15:04.750283 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:04.750266 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896035-wkzf6_odh-ai-gateway-infra(9ec13397-ce3a-4b29-a74a-b2e15ecba84d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" Nov 04 03:15:04.758448 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.758430 4784 scope.go:122] "RemoveContainer" containerID="6a9f21964dca78f0c96f1a96957789f1bb6cf21be92304af9cc5ecc05a570b1d" Nov 04 03:15:04.771710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:04.771686 4784 scope.go:122] "RemoveContainer" containerID="05b1e293bf15417ce31209e075f97ff1a79d7b86e108ee60c86036209562371f" Nov 04 03:15:05.759494 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:05.759465 4784 scope.go:122] "RemoveContainer" containerID="bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9" Nov 04 03:15:05.760125 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:05.759772 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896035-wkzf6_odh-ai-gateway-infra(9ec13397-ce3a-4b29-a74a-b2e15ecba84d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" Nov 04 03:15:05.761271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:05.761245 4784 scope.go:122] "RemoveContainer" containerID="d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6" Nov 04 03:15:05.761563 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:05.761539 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps_odh-ai-gateway-infra(524920e9-b313-4888-9fe3-c292c98b405a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" podUID="524920e9-b313-4888-9fe3-c292c98b405a" Nov 04 03:15:05.762813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:05.762796 4784 scope.go:122] "RemoveContainer" containerID="2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7" Nov 04 03:15:05.763025 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:05.763007 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt_odh-ai-gateway-infra(c28f67c9-86d4-43db-90fb-40252a0da44d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" Nov 04 03:15:06.401569 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:06.401521 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:15:15.153080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:15.152987 4784 scope.go:122] "RemoveContainer" containerID="d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9" Nov 04 03:15:15.153609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:15.153137 4784 scope.go:122] "RemoveContainer" containerID="2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7" Nov 04 03:15:15.861260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:15.861151 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerStarted","Data":"2e71ca99428071836798cebe324fc3d71d90acb553e5de6e54d9b29bbbf61028"} Nov 04 03:15:15.863228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:15.863203 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerStarted","Data":"94c27badcc614c06d78582ebb2d31640df61b94b3da4334eccaae1a8ef9c9a12"} Nov 04 03:15:16.153038 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.152950 4784 scope.go:122] "RemoveContainer" containerID="bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9" Nov 04 03:15:16.871971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.871932 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerStarted","Data":"3e38700b7aeec7826132f5b8182a7d56b5b023ed6b579ef37f3f98f3d5506357"} Nov 04 03:15:16.873953 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.873925 4784 generic.go:356] "Generic (PLEG): container finished" podID="e13f9282-abcb-487e-af52-e756e925714a" containerID="2e71ca99428071836798cebe324fc3d71d90acb553e5de6e54d9b29bbbf61028" exitCode=60 Nov 04 03:15:16.874110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.873976 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerDied","Data":"2e71ca99428071836798cebe324fc3d71d90acb553e5de6e54d9b29bbbf61028"} Nov 04 03:15:16.874110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.874036 4784 scope.go:122] "RemoveContainer" containerID="d60237febdf1014568964714ed648d5f6dc20d9071eae3cf64427d076a1d98d9" Nov 04 03:15:16.874385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.874369 4784 scope.go:122] "RemoveContainer" containerID="2e71ca99428071836798cebe324fc3d71d90acb553e5de6e54d9b29bbbf61028" Nov 04 03:15:16.874670 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:16.874652 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb_odh-ai-gateway-infra(e13f9282-abcb-487e-af52-e756e925714a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" podUID="e13f9282-abcb-487e-af52-e756e925714a" Nov 04 03:15:16.876086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.876066 4784 generic.go:356] "Generic (PLEG): container finished" podID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerID="94c27badcc614c06d78582ebb2d31640df61b94b3da4334eccaae1a8ef9c9a12" exitCode=60 Nov 04 03:15:16.876206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.876135 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerDied","Data":"94c27badcc614c06d78582ebb2d31640df61b94b3da4334eccaae1a8ef9c9a12"} Nov 04 03:15:16.876428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.876415 4784 scope.go:122] "RemoveContainer" containerID="94c27badcc614c06d78582ebb2d31640df61b94b3da4334eccaae1a8ef9c9a12" Nov 04 03:15:16.876611 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:16.876595 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt_odh-ai-gateway-infra(c28f67c9-86d4-43db-90fb-40252a0da44d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" Nov 04 03:15:16.887884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.887862 4784 scope.go:122] "RemoveContainer" containerID="2a6787f237c6baf1b52a1ddf201f2c21af8bacaad137ec9437b3e9962e8efbe7" Nov 04 03:15:16.990970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:16.990883 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:17.107687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.107647 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:17.153234 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.153205 4784 scope.go:122] "RemoveContainer" containerID="d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6" Nov 04 03:15:17.885996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.885962 4784 generic.go:356] "Generic (PLEG): container finished" podID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerID="3e38700b7aeec7826132f5b8182a7d56b5b023ed6b579ef37f3f98f3d5506357" exitCode=60 Nov 04 03:15:17.886538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.886037 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerDied","Data":"3e38700b7aeec7826132f5b8182a7d56b5b023ed6b579ef37f3f98f3d5506357"} Nov 04 03:15:17.886538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.886146 4784 scope.go:122] "RemoveContainer" containerID="bdb687f2a1b37ef684444f0fd34f371f613c08228439d8bc0e06e7ce408477e9" Nov 04 03:15:17.886661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.886563 4784 scope.go:122] "RemoveContainer" containerID="3e38700b7aeec7826132f5b8182a7d56b5b023ed6b579ef37f3f98f3d5506357" Nov 04 03:15:17.886871 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:17.886841 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896035-wkzf6_odh-ai-gateway-infra(9ec13397-ce3a-4b29-a74a-b2e15ecba84d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" Nov 04 03:15:17.891052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:17.891024 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerStarted","Data":"3c260b41c25471edded4c1b09f9d38cdd2fc0a1ce31651a5957dc65895ab901f"} Nov 04 03:15:18.022151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.022115 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:18.081524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.081493 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:18.119663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.119630 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:18.181892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.181860 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\" (UniqueName: \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\") pod \"e13f9282-abcb-487e-af52-e756e925714a\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " Nov 04 03:15:18.182080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.182001 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\") pod \"e13f9282-abcb-487e-af52-e756e925714a\" (UID: \"e13f9282-abcb-487e-af52-e756e925714a\") " Nov 04 03:15:18.182080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.182058 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\" (UniqueName: \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\") pod \"c28f67c9-86d4-43db-90fb-40252a0da44d\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " Nov 04 03:15:18.182080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.182078 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\") pod \"c28f67c9-86d4-43db-90fb-40252a0da44d\" (UID: \"c28f67c9-86d4-43db-90fb-40252a0da44d\") " Nov 04 03:15:18.182486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.182451 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca" pod "e13f9282-abcb-487e-af52-e756e925714a" (UID: "e13f9282-abcb-487e-af52-e756e925714a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:15:18.182486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.182471 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca" pod "c28f67c9-86d4-43db-90fb-40252a0da44d" (UID: "c28f67c9-86d4-43db-90fb-40252a0da44d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:15:18.184425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.184405 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7" pod "c28f67c9-86d4-43db-90fb-40252a0da44d" (UID: "c28f67c9-86d4-43db-90fb-40252a0da44d"). InnerVolumeSpecName "kube-api-access-dfxd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:15:18.184520 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.184477 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx" pod "e13f9282-abcb-487e-af52-e756e925714a" (UID: "e13f9282-abcb-487e-af52-e756e925714a"). InnerVolumeSpecName "kube-api-access-slfxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:15:18.283651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.283613 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxd7\" (UniqueName: \"kubernetes.io/projected/c28f67c9-86d4-43db-90fb-40252a0da44d-kube-api-access-dfxd7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:18.283651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.283642 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28f67c9-86d4-43db-90fb-40252a0da44d-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:18.283651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.283653 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfxx\" (UniqueName: \"kubernetes.io/projected/e13f9282-abcb-487e-af52-e756e925714a-kube-api-access-slfxx\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:18.283651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.283662 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13f9282-abcb-487e-af52-e756e925714a-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:18.905931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.905895 4784 generic.go:356] "Generic (PLEG): container finished" podID="524920e9-b313-4888-9fe3-c292c98b405a" containerID="3c260b41c25471edded4c1b09f9d38cdd2fc0a1ce31651a5957dc65895ab901f" exitCode=60 Nov 04 03:15:18.906385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.905957 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerDied","Data":"3c260b41c25471edded4c1b09f9d38cdd2fc0a1ce31651a5957dc65895ab901f"} Nov 04 03:15:18.906385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.906016 4784 scope.go:122] "RemoveContainer" containerID="d35592f80dd568851c9537d1f633f0eb57f5bd2ddbaad177cd546356f30577f6" Nov 04 03:15:18.906517 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.906497 4784 scope.go:122] "RemoveContainer" containerID="3c260b41c25471edded4c1b09f9d38cdd2fc0a1ce31651a5957dc65895ab901f" Nov 04 03:15:18.906879 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:18.906849 4784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps_odh-ai-gateway-infra(524920e9-b313-4888-9fe3-c292c98b405a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" podUID="524920e9-b313-4888-9fe3-c292c98b405a" Nov 04 03:15:18.907687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.907663 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" event={"ID":"e13f9282-abcb-487e-af52-e756e925714a","Type":"ContainerDied","Data":"ed7049383c6b06fed7238a506b2350f8e1561539082b8ba396a723fadb4feca7"} Nov 04 03:15:18.907810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.907669 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb" Nov 04 03:15:18.909241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.909220 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" event={"ID":"c28f67c9-86d4-43db-90fb-40252a0da44d","Type":"ContainerDied","Data":"75dbfdff492b218554ae45017af6a9a7cf6b3a468aa9830069b602bcc71910ee"} Nov 04 03:15:18.909356 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.909243 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt" Nov 04 03:15:18.919669 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.919649 4784 scope.go:122] "RemoveContainer" containerID="2e71ca99428071836798cebe324fc3d71d90acb553e5de6e54d9b29bbbf61028" Nov 04 03:15:18.932544 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:18.932517 4784 scope.go:122] "RemoveContainer" containerID="94c27badcc614c06d78582ebb2d31640df61b94b3da4334eccaae1a8ef9c9a12" Nov 04 03:15:19.060206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.060162 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:19.092621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.092591 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\") pod \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " Nov 04 03:15:19.092821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.092758 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\" (UniqueName: \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\") pod \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\" (UID: \"9ec13397-ce3a-4b29-a74a-b2e15ecba84d\") " Nov 04 03:15:19.092962 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.092937 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca" pod "9ec13397-ce3a-4b29-a74a-b2e15ecba84d" (UID: "9ec13397-ce3a-4b29-a74a-b2e15ecba84d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:15:19.093170 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.093148 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:19.095131 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.095106 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn" pod "9ec13397-ce3a-4b29-a74a-b2e15ecba84d" (UID: "9ec13397-ce3a-4b29-a74a-b2e15ecba84d"). InnerVolumeSpecName "kube-api-access-dcctn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:15:19.136076 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.136042 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:19.194402 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.194369 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcctn\" (UniqueName: \"kubernetes.io/projected/9ec13397-ce3a-4b29-a74a-b2e15ecba84d-kube-api-access-dcctn\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:19.265269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.265234 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:19.269114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.269081 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-095081-29896035-jmghb"] Nov 04 03:15:19.393425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.393383 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:19.407641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.407594 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a0bcbc-29896035-v2szt"] Nov 04 03:15:19.917091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.917047 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" Nov 04 03:15:19.917091 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.917082 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6" event={"ID":"9ec13397-ce3a-4b29-a74a-b2e15ecba84d","Type":"ContainerDied","Data":"0874f2f3b2b7f0215b3497b8955aa8a6466be05aa5ab425111f97dae0584fd10"} Nov 04 03:15:19.917667 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:19.917142 4784 scope.go:122] "RemoveContainer" containerID="3e38700b7aeec7826132f5b8182a7d56b5b023ed6b579ef37f3f98f3d5506357" Nov 04 03:15:20.071464 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.071434 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:20.088690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.088662 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:20.093336 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.093309 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896035-wkzf6"] Nov 04 03:15:20.106056 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.106029 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\" (UniqueName: \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\") pod \"524920e9-b313-4888-9fe3-c292c98b405a\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " Nov 04 03:15:20.106170 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.106070 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\") pod \"524920e9-b313-4888-9fe3-c292c98b405a\" (UID: \"524920e9-b313-4888-9fe3-c292c98b405a\") " Nov 04 03:15:20.106483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.106461 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca" pod "524920e9-b313-4888-9fe3-c292c98b405a" (UID: "524920e9-b313-4888-9fe3-c292c98b405a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:15:20.108332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.108311 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26" pod "524920e9-b313-4888-9fe3-c292c98b405a" (UID: "524920e9-b313-4888-9fe3-c292c98b405a"). InnerVolumeSpecName "kube-api-access-c2v26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:15:20.208165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.208064 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2v26\" (UniqueName: \"kubernetes.io/projected/524920e9-b313-4888-9fe3-c292c98b405a-kube-api-access-c2v26\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:20.208165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.208107 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/524920e9-b313-4888-9fe3-c292c98b405a-maas-api-service-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:20.931398 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.931360 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" Nov 04 03:15:20.931909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.931405 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps" event={"ID":"524920e9-b313-4888-9fe3-c292c98b405a","Type":"ContainerDied","Data":"f8eb633dee21767ef6999dec53ba35db743b0f6bbb52d2d924cad56eeae64195"} Nov 04 03:15:20.931909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:20.931499 4784 scope.go:122] "RemoveContainer" containerID="3c260b41c25471edded4c1b09f9d38cdd2fc0a1ce31651a5957dc65895ab901f" Nov 04 03:15:21.102999 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.102965 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:21.109035 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.108999 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896035-hlsps"] Nov 04 03:15:21.161207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.161165 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524920e9-b313-4888-9fe3-c292c98b405a" path="/var/lib/kubelet/pods/524920e9-b313-4888-9fe3-c292c98b405a/volumes" Nov 04 03:15:21.161627 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.161612 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" path="/var/lib/kubelet/pods/9ec13397-ce3a-4b29-a74a-b2e15ecba84d/volumes" Nov 04 03:15:21.161968 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.161956 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" path="/var/lib/kubelet/pods/c28f67c9-86d4-43db-90fb-40252a0da44d/volumes" Nov 04 03:15:21.162622 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:21.162608 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13f9282-abcb-487e-af52-e756e925714a" path="/var/lib/kubelet/pods/e13f9282-abcb-487e-af52-e756e925714a/volumes" Nov 04 03:15:36.401613 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:15:36.401567 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:15:36.484228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:36.484147 4784 scope.go:122] "RemoveContainer" containerID="655124c487d1ced04bb29744d3b26da2abefc858ddd83fd6aaf5a6a446076cf1" Nov 04 03:15:36.495852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:36.495825 4784 scope.go:122] "RemoveContainer" containerID="9fd08b09b0ebbc2e1e78801cd491ac64e85fb14bc33c40ace2213d47be088699" Nov 04 03:15:36.505444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:36.505422 4784 scope.go:122] "RemoveContainer" containerID="7b9aa6cfd50fc0b2afd6cf17458013de9e339a91ae3a340d22d4817a7164a5b3" Nov 04 03:15:38.051367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:38.051337 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz"] Nov 04 03:15:38.104619 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:38.104584 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65a27aa-68cvz"] Nov 04 03:15:39.162661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:39.162614 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8860e9c6-8212-4eff-a6cf-228624c4fd3c" path="/var/lib/kubelet/pods/8860e9c6-8212-4eff-a6cf-228624c4fd3c/volumes" Nov 04 03:15:46.050160 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:46.050120 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6"] Nov 04 03:15:46.106805 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:46.102872 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f5ca53-8dpd6"] Nov 04 03:15:47.161435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:47.161400 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b37ae8d-f677-4777-a4f8-1ef21992c30f" path="/var/lib/kubelet/pods/7b37ae8d-f677-4777-a4f8-1ef21992c30f/volumes" Nov 04 03:15:49.050657 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.050610 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s"] Nov 04 03:15:49.101668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.101628 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0a837b-r245s"] Nov 04 03:15:49.160344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.160304 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbeb4fc-1114-4c38-88a6-d5b404d2d46b" path="/var/lib/kubelet/pods/efbeb4fc-1114-4c38-88a6-d5b404d2d46b/volumes" Nov 04 03:15:49.392575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.392470 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs"] Nov 04 03:15:49.393111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393088 4784 cpu_manager.go:414] "containerMap: removing container" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393112 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393130 4784 cpu_manager.go:414] "containerMap: removing container" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393137 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393164 4784 cpu_manager.go:414] "containerMap: removing container" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393173 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393208 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393215 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393227 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393233 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393243 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393249 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393267 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393274 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393289 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393297 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393317 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393325 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393335 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393342 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393358 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393365 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393377 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393384 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393530 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393542 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393553 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393563 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28f67c9-86d4-43db-90fb-40252a0da44d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393573 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393582 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393590 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393598 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13f9282-abcb-487e-af52-e756e925714a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393609 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393618 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.393827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.393631 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="524920e9-b313-4888-9fe3-c292c98b405a" containerName="cleanup" Nov 04 03:15:49.399103 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.399078 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.453979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.453937 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs"] Nov 04 03:15:49.485223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.482662 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.485223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.482811 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.485223 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.482851 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.500854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.500824 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:15:49.500854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.500822 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:15:49.501114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.500873 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:15:49.503512 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.503495 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:15:49.583749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.583712 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.584332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.583780 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.584332 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.583804 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.584481 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.584459 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.586568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.586543 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.646710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.646627 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\") pod \"maas-api-revoke-sub-16d69779-w2mvs\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:49.712936 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:49.712901 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:50.110542 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:15:50.110514 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afed593_cf42_4024_a89f_a8478f85248d.slice/crio-563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c WatchSource:0}: Error finding container 563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c: Status 404 returned error can't find the container with id 563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c Nov 04 03:15:50.111015 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:50.110975 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs"] Nov 04 03:15:50.148646 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:50.148615 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" event={"ID":"8afed593-cf42-4024-a89f-a8478f85248d","Type":"ContainerStarted","Data":"563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c"} Nov 04 03:15:51.158449 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:51.158416 4784 generic.go:356] "Generic (PLEG): container finished" podID="8afed593-cf42-4024-a89f-a8478f85248d" containerID="41fcc018dc7b62c8288e1b35209d6b0c23b2a46778cb07cba368188d66567947" exitCode=0 Nov 04 03:15:51.160779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:51.160750 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" event={"ID":"8afed593-cf42-4024-a89f-a8478f85248d","Type":"ContainerDied","Data":"41fcc018dc7b62c8288e1b35209d6b0c23b2a46778cb07cba368188d66567947"} Nov 04 03:15:52.337396 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.337370 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:52.418120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.418030 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\") pod \"8afed593-cf42-4024-a89f-a8478f85248d\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " Nov 04 03:15:52.418320 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.418136 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\") pod \"8afed593-cf42-4024-a89f-a8478f85248d\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " Nov 04 03:15:52.418320 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.418270 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\") pod \"8afed593-cf42-4024-a89f-a8478f85248d\" (UID: \"8afed593-cf42-4024-a89f-a8478f85248d\") " Nov 04 03:15:52.418526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.418498 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca" pod "8afed593-cf42-4024-a89f-a8478f85248d" (UID: "8afed593-cf42-4024-a89f-a8478f85248d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:15:52.419051 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.419022 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:52.420562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.420532 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca" pod "8afed593-cf42-4024-a89f-a8478f85248d" (UID: "8afed593-cf42-4024-a89f-a8478f85248d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:15:52.420676 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.420650 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k" pod "8afed593-cf42-4024-a89f-a8478f85248d" (UID: "8afed593-cf42-4024-a89f-a8478f85248d"). InnerVolumeSpecName "kube-api-access-kfs9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:15:52.520363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.520326 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/8afed593-cf42-4024-a89f-a8478f85248d-kube-api-access-kfs9k\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:52.520363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:52.520357 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afed593-cf42-4024-a89f-a8478f85248d-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:15:53.172798 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:53.172771 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" Nov 04 03:15:53.172798 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:53.172789 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs" event={"ID":"8afed593-cf42-4024-a89f-a8478f85248d","Type":"ContainerDied","Data":"563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c"} Nov 04 03:15:53.172993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:15:53.172835 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563b620f2dd55782da29abc9e26e3b2cde21e3700be3275ffd7bd28f66ac964c" Nov 04 03:16:02.514916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.514875 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr"] Nov 04 03:16:02.515539 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.515522 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8afed593-cf42-4024-a89f-a8478f85248d" containerName="revoke-keys" Nov 04 03:16:02.515587 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.515541 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afed593-cf42-4024-a89f-a8478f85248d" containerName="revoke-keys" Nov 04 03:16:02.515647 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.515638 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec13397-ce3a-4b29-a74a-b2e15ecba84d" containerName="cleanup" Nov 04 03:16:02.515687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.515652 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afed593-cf42-4024-a89f-a8478f85248d" containerName="revoke-keys" Nov 04 03:16:02.519108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.519091 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.577142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.577094 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr"] Nov 04 03:16:02.623965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.623931 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:16:02.623965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.623962 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:16:02.636836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.636809 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:16:02.637372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.637354 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:16:02.665435 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.665400 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtq8\" (UniqueName: \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.665604 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.665526 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.665604 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.665555 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.766646 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.766562 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtq8\" (UniqueName: \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.766813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.766645 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.766813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.766666 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.767337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.767316 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.769276 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.769256 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.829501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.829467 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtq8\" (UniqueName: \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\") pod \"maas-api-revoke-sub-f2953abc-kxcwr\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:02.833268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:02.833245 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:03.248695 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:03.237760 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr"] Nov 04 03:16:03.262892 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:16:03.260543 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cce5c5_8ab6_4cad_924c_974dd29bc5e2.slice/crio-d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c WatchSource:0}: Error finding container d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c: Status 404 returned error can't find the container with id d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c Nov 04 03:16:04.053369 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:04.053330 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf"] Nov 04 03:16:04.103883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:04.103845 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fedd328-9ttjf"] Nov 04 03:16:04.281298 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:04.281260 4784 generic.go:356] "Generic (PLEG): container finished" podID="f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" containerID="1764e675eeae093581dcbd11a73454f7805e9f6fa920be7776a719d3b68f8433" exitCode=0 Nov 04 03:16:04.281478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:04.281318 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" event={"ID":"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2","Type":"ContainerDied","Data":"1764e675eeae093581dcbd11a73454f7805e9f6fa920be7776a719d3b68f8433"} Nov 04 03:16:04.281478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:04.281378 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" event={"ID":"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2","Type":"ContainerStarted","Data":"d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c"} Nov 04 03:16:05.160940 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.160906 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0129b8-9f5c-4278-a80f-d5068550ffa9" path="/var/lib/kubelet/pods/ad0129b8-9f5c-4278-a80f-d5068550ffa9/volumes" Nov 04 03:16:05.446428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.446401 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:05.601689 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.601650 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\") pod \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " Nov 04 03:16:05.601916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.601715 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\" (UniqueName: \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\") pod \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " Nov 04 03:16:05.601916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.601887 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\") pod \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\" (UID: \"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2\") " Nov 04 03:16:05.602527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.602493 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca" pod "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" (UID: "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:16:05.604063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.604038 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca" pod "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" (UID: "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:16:05.604280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.604262 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8" pod "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" (UID: "f8cce5c5-8ab6-4cad-924c-974dd29bc5e2"). InnerVolumeSpecName "kube-api-access-4vtq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:05.703712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.703623 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:05.703712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.703656 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtq8\" (UniqueName: \"kubernetes.io/projected/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-kube-api-access-4vtq8\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:05.703712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:05.703668 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:06.298630 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:06.298596 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" Nov 04 03:16:06.299041 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:06.298602 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr" event={"ID":"f8cce5c5-8ab6-4cad-924c-974dd29bc5e2","Type":"ContainerDied","Data":"d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c"} Nov 04 03:16:06.299041 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:06.298730 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d692aade77113cb78cf1b35d4b3a217eb61d2bfe0ba2090d51e84bd119d68b5c" Nov 04 03:16:06.401428 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:16:06.401385 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:16:07.049607 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:07.049569 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg"] Nov 04 03:16:07.100165 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:07.100110 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1233a27-dq8tg"] Nov 04 03:16:07.160704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:07.160672 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6714fd46-a2b6-4b05-9eac-2ad039feef14" path="/var/lib/kubelet/pods/6714fd46-a2b6-4b05-9eac-2ad039feef14/volumes" Nov 04 03:16:12.050059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:12.050013 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd"] Nov 04 03:16:12.100806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:12.100767 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57edbdda-gv9bd"] Nov 04 03:16:13.163763 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:13.163718 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16073ce3-a672-4271-909f-08f22bbb07a8" path="/var/lib/kubelet/pods/16073ce3-a672-4271-909f-08f22bbb07a8/volumes" Nov 04 03:16:14.057922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:14.057211 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq"] Nov 04 03:16:14.105921 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:14.105884 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b6610-f7kxq"] Nov 04 03:16:15.161206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:15.161154 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4629ecf-6050-436a-840b-3e10c0e659c6" path="/var/lib/kubelet/pods/d4629ecf-6050-436a-840b-3e10c0e659c6/volumes" Nov 04 03:16:23.049990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:23.049952 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5"] Nov 04 03:16:23.100488 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:23.100455 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f53f08d-dc5r5"] Nov 04 03:16:23.161099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:23.161062 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0534750f-a52f-4dcb-a291-747edbb2ac5c" path="/var/lib/kubelet/pods/0534750f-a52f-4dcb-a291-747edbb2ac5c/volumes" Nov 04 03:16:28.049806 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:28.049764 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm"] Nov 04 03:16:28.100821 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:28.100780 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f10719-wkrrm"] Nov 04 03:16:28.822516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:28.822480 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 03:16:28.822826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:28.822739 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" containerID="cri-o://851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd" gracePeriod=30 Nov 04 03:16:29.160795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:29.160708 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7dbec3-f271-4601-b2a8-15ba25d6b744" path="/var/lib/kubelet/pods/aa7dbec3-f271-4601-b2a8-15ba25d6b744/volumes" Nov 04 03:16:34.118444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.118370 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 03:16:34.247085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247043 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247108 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247170 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247292 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247368 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247402 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247434 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247469 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247517 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\") pod \"6d0718ab-45bb-49ce-8d9f-216970d57496\" (UID: \"6d0718ab-45bb-49ce-8d9f-216970d57496\") " Nov 04 03:16:34.247824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247648 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:34.247824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247688 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:34.247824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247780 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:34.248030 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247902 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:34.248030 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.247982 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:16:34.248308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.248149 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.248308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.248167 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.248308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.248182 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.248308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.248219 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.248308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.248228 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0718ab-45bb-49ce-8d9f-216970d57496-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.249854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.249827 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:34.249997 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.249857 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "kube-api-access-glqjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:34.250079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.250059 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:34.250133 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.250117 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo" pod "6d0718ab-45bb-49ce-8d9f-216970d57496" (UID: "6d0718ab-45bb-49ce-8d9f-216970d57496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:16:34.349643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.349596 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.349643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.349629 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqjv\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-kube-api-access-glqjv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.349643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.349644 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.349643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.349656 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0718ab-45bb-49ce-8d9f-216970d57496-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:34.502491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.502451 4784 generic.go:356] "Generic (PLEG): container finished" podID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerID="851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd" exitCode=0 Nov 04 03:16:34.502710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.502513 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" Nov 04 03:16:34.502710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.502530 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerDied","Data":"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd"} Nov 04 03:16:34.502710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.502588 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq" event={"ID":"6d0718ab-45bb-49ce-8d9f-216970d57496","Type":"ContainerDied","Data":"2facf34394c49823d57a0a4e29c9f6e78c797c155117dfb9465baf8bcc602520"} Nov 04 03:16:34.502710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.502600 4784 scope.go:122] "RemoveContainer" containerID="851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd" Nov 04 03:16:34.514839 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.514810 4784 scope.go:122] "RemoveContainer" containerID="136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972" Nov 04 03:16:34.526764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.526741 4784 scope.go:122] "RemoveContainer" containerID="851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd" Nov 04 03:16:34.527078 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:16:34.527055 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd\": container with ID starting with 851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd not found: ID does not exist" containerID="851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd" Nov 04 03:16:34.527151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.527087 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd"} err="failed to get container status \"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd\": rpc error: code = NotFound desc = could not find container \"851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd\": container with ID starting with 851b97e4133847bc351026fb57ce73283b347d13a21515263f672c0f893fe0cd not found: ID does not exist" Nov 04 03:16:34.527151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.527105 4784 scope.go:122] "RemoveContainer" containerID="136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972" Nov 04 03:16:34.527436 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:16:34.527415 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972\": container with ID starting with 136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972 not found: ID does not exist" containerID="136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972" Nov 04 03:16:34.527494 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.527442 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972"} err="failed to get container status \"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972\": rpc error: code = NotFound desc = could not find container \"136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972\": container with ID starting with 136720e89b361a405b5e24dc3f46da067101c52c236359bbf9154bf93cc35972 not found: ID does not exist" Nov 04 03:16:34.673075 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.673043 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 03:16:34.676856 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:34.676831 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-v2tpq"] Nov 04 03:16:35.166220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:35.163983 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" path="/var/lib/kubelet/pods/6d0718ab-45bb-49ce-8d9f-216970d57496/volumes" Nov 04 03:16:36.401364 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:16:36.401319 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:16:36.559852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:36.559814 4784 scope.go:122] "RemoveContainer" containerID="cd237ddcd9e61d305e47c84901743ae3e6283bf5032e1dcac73ff774830fe46b" Nov 04 03:16:44.601507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:44.601406 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:16:44.712328 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:44.712287 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:16:49.602428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.602376 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" podUID="9d2eadc5-f33e-4517-831d-adfd898c068e" containerName="payload-processing" containerID="cri-o://de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07" gracePeriod=25 Nov 04 03:16:49.713308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.713242 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" containerID="cri-o://62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be" gracePeriod=25 Nov 04 03:16:49.872900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.872824 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:16:49.992153 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.992119 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\") pod \"9d2eadc5-f33e-4517-831d-adfd898c068e\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " Nov 04 03:16:49.992410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.992303 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\" (UniqueName: \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\") pod \"9d2eadc5-f33e-4517-831d-adfd898c068e\" (UID: \"9d2eadc5-f33e-4517-831d-adfd898c068e\") " Nov 04 03:16:49.993049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.993004 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume" pod "9d2eadc5-f33e-4517-831d-adfd898c068e" (UID: "9d2eadc5-f33e-4517-831d-adfd898c068e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:16:49.995502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:49.995476 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c" pod "9d2eadc5-f33e-4517-831d-adfd898c068e" (UID: "9d2eadc5-f33e-4517-831d-adfd898c068e"). InnerVolumeSpecName "kube-api-access-t8h7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:50.093621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.093576 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8h7c\" (UniqueName: \"kubernetes.io/projected/9d2eadc5-f33e-4517-831d-adfd898c068e-kube-api-access-t8h7c\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:50.093621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.093614 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2eadc5-f33e-4517-831d-adfd898c068e-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:50.624976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.624945 4784 generic.go:356] "Generic (PLEG): container finished" podID="9d2eadc5-f33e-4517-831d-adfd898c068e" containerID="de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07" exitCode=0 Nov 04 03:16:50.625504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.624992 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" Nov 04 03:16:50.625504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.624998 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" event={"ID":"9d2eadc5-f33e-4517-831d-adfd898c068e","Type":"ContainerDied","Data":"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07"} Nov 04 03:16:50.625504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.625043 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg" event={"ID":"9d2eadc5-f33e-4517-831d-adfd898c068e","Type":"ContainerDied","Data":"f436ddbd33f09e8c8135e19f1ec06cf8155e177f30ab76c31e6490f55e7f15ee"} Nov 04 03:16:50.625504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.625066 4784 scope.go:122] "RemoveContainer" containerID="de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07" Nov 04 03:16:50.636286 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.636261 4784 scope.go:122] "RemoveContainer" containerID="de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07" Nov 04 03:16:50.636631 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:16:50.636606 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07\": container with ID starting with de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07 not found: ID does not exist" containerID="de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07" Nov 04 03:16:50.636702 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.636641 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07"} err="failed to get container status \"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07\": rpc error: code = NotFound desc = could not find container \"de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07\": container with ID starting with de6bbd4fadba69da9b22a5ae938a17945bb7dd5d97f9463b1e7710e55e1aec07 not found: ID does not exist" Nov 04 03:16:50.793960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.793920 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:16:50.845018 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:50.844958 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db5ef5-657546bd74-dh9pg"] Nov 04 03:16:51.160889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:51.160853 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2eadc5-f33e-4517-831d-adfd898c068e" path="/var/lib/kubelet/pods/9d2eadc5-f33e-4517-831d-adfd898c068e/volumes" Nov 04 03:16:51.692991 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:16:51.692951 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:17:00.559705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.559665 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 03:17:00.560240 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.559936 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" containerID="cri-o://461550caafa986b2ec9eb85ec794428a77fa99019112cac2131f2efe9c4d944c" gracePeriod=30 Nov 04 03:17:00.740577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.740517 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 03:17:00.740824 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.740780 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" containerID="cri-o://6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c" gracePeriod=30 Nov 04 03:17:00.743341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.743310 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 03:17:00.743815 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:00.743783 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" containerID="cri-o://bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4" gracePeriod=30 Nov 04 03:17:01.693448 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:01.693409 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:17:02.337157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:02.337115 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xr2tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:17:02.337376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:02.337226 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:03.041198 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.041158 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 03:17:03.041616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.041422 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" containerID="cri-o://319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9" gracePeriod=30 Nov 04 03:17:03.343853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.343823 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 03:17:03.376155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376114 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376169 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376325 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376415 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376462 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376525 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376514 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376604 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\") pod \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\" (UID: \"80054ecc-72b0-4405-8f2c-2c8c283bc3f3\") " Nov 04 03:17:03.376781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.376749 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:03.377364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.377321 4784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-ca-configmap\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.379349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379313 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:03.379516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379314 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "kube-api-access-hft8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:03.379516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379501 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:03.379682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379599 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:03.379682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379626 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:03.379757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.379725 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts" pod "80054ecc-72b0-4405-8f2c-2c8c283bc3f3" (UID: "80054ecc-72b0-4405-8f2c-2c8c283bc3f3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:03.478979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.478936 4784 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-local-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.478979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.478982 4784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-csr-dns-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.479303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.478994 4784 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-cacerts\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.479303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.479005 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.479303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.479015 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hft8t\" (UniqueName: \"kubernetes.io/projected/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-kube-api-access-hft8t\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.479303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.479026 4784 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80054ecc-72b0-4405-8f2c-2c8c283bc3f3-istio-kubeconfig\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.726975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.726938 4784 generic.go:356] "Generic (PLEG): container finished" podID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerID="319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9" exitCode=0 Nov 04 03:17:03.727169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.726985 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" Nov 04 03:17:03.727169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.727012 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" event={"ID":"80054ecc-72b0-4405-8f2c-2c8c283bc3f3","Type":"ContainerDied","Data":"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9"} Nov 04 03:17:03.727169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.727064 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m" event={"ID":"80054ecc-72b0-4405-8f2c-2c8c283bc3f3","Type":"ContainerDied","Data":"4c19831d480139dfbc85e2f8948b3578d99628fbb6fb84abe09210fa24657be2"} Nov 04 03:17:03.727169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.727092 4784 scope.go:122] "RemoveContainer" containerID="319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9" Nov 04 03:17:03.739397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.739366 4784 scope.go:122] "RemoveContainer" containerID="319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9" Nov 04 03:17:03.739738 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:03.739713 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9\": container with ID starting with 319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9 not found: ID does not exist" containerID="319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9" Nov 04 03:17:03.739829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.739747 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9"} err="failed to get container status \"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9\": rpc error: code = NotFound desc = could not find container \"319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9\": container with ID starting with 319cbb2b58ba9ebd7f89fd6e5d99c7083a8cee3839dd8f3528c63f22678ccbc9 not found: ID does not exist" Nov 04 03:17:03.897080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.897041 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 03:17:03.903538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:03.903505 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t6h4m"] Nov 04 03:17:05.160086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.160052 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" path="/var/lib/kubelet/pods/80054ecc-72b0-4405-8f2c-2c8c283bc3f3/volumes" Nov 04 03:17:05.459444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.459406 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:17:05.459677 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.459651 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" containerID="cri-o://353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531" gracePeriod=30 Nov 04 03:17:05.722162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.722054 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-zhmt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:17:05.722162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.722128 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:05.770483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.770412 4784 generic.go:356] "Generic (PLEG): container finished" podID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerID="461550caafa986b2ec9eb85ec794428a77fa99019112cac2131f2efe9c4d944c" exitCode=0 Nov 04 03:17:05.770751 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.770622 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerDied","Data":"461550caafa986b2ec9eb85ec794428a77fa99019112cac2131f2efe9c4d944c"} Nov 04 03:17:05.770751 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.770730 4784 scope.go:122] "RemoveContainer" containerID="cdb281b33af22ea6baecae9164d34defff666dc1bd176aeff6a2cf7c0688268d" Nov 04 03:17:05.928095 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.928065 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 03:17:05.942921 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:05.942895 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 03:17:06.007235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007122 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.007235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007173 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007220 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007249 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007288 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007342 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007373 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007442 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007513 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007540 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007550 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.007575 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007572 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007619 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007656 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007680 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007706 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007763 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007821 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007854 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\") pod \"e1afcda6-7342-4238-a55b-14b7aee67b61\" (UID: \"e1afcda6-7342-4238-a55b-14b7aee67b61\") " Nov 04 03:17:06.008079 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007935 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\") pod \"77b0097c-5696-43a3-a6cd-74ff8d85897f\" (UID: \"77b0097c-5696-43a3-a6cd-74ff8d85897f\") " Nov 04 03:17:06.008550 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008496 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.008550 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008518 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.008749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.007815 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:06.008749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008004 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.008749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008002 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.008749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008107 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:06.008749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008455 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.009151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008805 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.009151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.008918 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.009151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.009003 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.010636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.010609 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "kube-api-access-9j7m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.010758 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.010647 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.011271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011251 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.011339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011318 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:06.011524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011496 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.011749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011734 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token" pod "e1afcda6-7342-4238-a55b-14b7aee67b61" (UID: "e1afcda6-7342-4238-a55b-14b7aee67b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.011810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011774 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "kube-api-access-wrlmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.011810 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.011795 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo" pod "77b0097c-5696-43a3-a6cd-74ff8d85897f" (UID: "77b0097c-5696-43a3-a6cd-74ff8d85897f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:06.109303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109265 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrlmc\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-kube-api-access-wrlmc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109300 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j7m6\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-kube-api-access-9j7m6\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109311 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109322 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109332 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109341 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109351 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109359 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109368 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109375 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109385 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109394 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1afcda6-7342-4238-a55b-14b7aee67b61-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109401 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0097c-5696-43a3-a6cd-74ff8d85897f-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109410 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1afcda6-7342-4238-a55b-14b7aee67b61-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109419 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0097c-5696-43a3-a6cd-74ff8d85897f-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.109569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.109427 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0097c-5696-43a3-a6cd-74ff8d85897f-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.404242 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:06.404202 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:17:06.417933 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.417907 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 03:17:06.514058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514022 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514066 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514137 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514223 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514255 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514278 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514347 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514385 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514408 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.514628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514420 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.514628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514448 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\") pod \"8040dd82-b66b-4c36-ad01-c466a63ffc11\" (UID: \"8040dd82-b66b-4c36-ad01-c466a63ffc11\") " Nov 04 03:17:06.514892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514706 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.514892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514742 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.514983 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514930 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.514983 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514955 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.514983 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514968 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.515140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514982 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.515140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.514988 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:06.516788 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.516765 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "kube-api-access-htrwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.516788 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.516776 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.517129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.517105 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:06.517129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.517111 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy" pod "8040dd82-b66b-4c36-ad01-c466a63ffc11" (UID: "8040dd82-b66b-4c36-ad01-c466a63ffc11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.616504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.616469 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.616504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.616502 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.616504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.616512 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8040dd82-b66b-4c36-ad01-c466a63ffc11-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.616889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.616522 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8040dd82-b66b-4c36-ad01-c466a63ffc11-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.616889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.616532 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrwf\" (UniqueName: \"kubernetes.io/projected/8040dd82-b66b-4c36-ad01-c466a63ffc11-kube-api-access-htrwf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.779975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.779940 4784 generic.go:356] "Generic (PLEG): container finished" podID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerID="6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c" exitCode=0 Nov 04 03:17:06.780233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.780016 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" Nov 04 03:17:06.780233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.780022 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerDied","Data":"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c"} Nov 04 03:17:06.780233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.780112 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8" event={"ID":"e1afcda6-7342-4238-a55b-14b7aee67b61","Type":"ContainerDied","Data":"5ead32480ca9ca74f1295c39926a42eb1574c9ca6ae028f12eb94a37600ea4ee"} Nov 04 03:17:06.780233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.780136 4784 scope.go:122] "RemoveContainer" containerID="6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c" Nov 04 03:17:06.781887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.781694 4784 generic.go:356] "Generic (PLEG): container finished" podID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerID="bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4" exitCode=0 Nov 04 03:17:06.781887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.781751 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" Nov 04 03:17:06.781887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.781790 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" event={"ID":"8040dd82-b66b-4c36-ad01-c466a63ffc11","Type":"ContainerDied","Data":"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4"} Nov 04 03:17:06.781887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.781857 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj" event={"ID":"8040dd82-b66b-4c36-ad01-c466a63ffc11","Type":"ContainerDied","Data":"547f6d0a34bc8a77737c246a5999999295476fbb83e7fd9e2d9287b63b0643ae"} Nov 04 03:17:06.783546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.783521 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" event={"ID":"77b0097c-5696-43a3-a6cd-74ff8d85897f","Type":"ContainerDied","Data":"af60dbd489457aa43fcda35129f04e49a129f8fd5c53a3c99eca7d7b394d5803"} Nov 04 03:17:06.783653 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.783578 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw" Nov 04 03:17:06.793678 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.793656 4784 scope.go:122] "RemoveContainer" containerID="569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c" Nov 04 03:17:06.804610 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.804586 4784 scope.go:122] "RemoveContainer" containerID="6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c" Nov 04 03:17:06.804936 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:06.804916 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c\": container with ID starting with 6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c not found: ID does not exist" containerID="6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c" Nov 04 03:17:06.804994 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.804948 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c"} err="failed to get container status \"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c\": rpc error: code = NotFound desc = could not find container \"6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c\": container with ID starting with 6353d836f419bac3eb3a69ff3272ae6731576ac64bc77ca0c3e6cb976a3a4c6c not found: ID does not exist" Nov 04 03:17:06.804994 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.804967 4784 scope.go:122] "RemoveContainer" containerID="569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c" Nov 04 03:17:06.805289 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:06.805269 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c\": container with ID starting with 569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c not found: ID does not exist" containerID="569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c" Nov 04 03:17:06.805372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.805294 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c"} err="failed to get container status \"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c\": rpc error: code = NotFound desc = could not find container \"569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c\": container with ID starting with 569b2c94df887b64361e7eb45bf816cc773e5f58a684eb0a6b6a366ca10d347c not found: ID does not exist" Nov 04 03:17:06.805372 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.805310 4784 scope.go:122] "RemoveContainer" containerID="bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4" Nov 04 03:17:06.815128 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.815108 4784 scope.go:122] "RemoveContainer" containerID="bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4" Nov 04 03:17:06.815461 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:06.815443 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4\": container with ID starting with bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4 not found: ID does not exist" containerID="bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4" Nov 04 03:17:06.815548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.815475 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4"} err="failed to get container status \"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4\": rpc error: code = NotFound desc = could not find container \"bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4\": container with ID starting with bc55e3c99bd5afd2de8373d88f8d49b1791e7ff2f22f810b55b6d446cc8ac3a4 not found: ID does not exist" Nov 04 03:17:06.815548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.815500 4784 scope.go:122] "RemoveContainer" containerID="461550caafa986b2ec9eb85ec794428a77fa99019112cac2131f2efe9c4d944c" Nov 04 03:17:06.949539 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:06.949501 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 03:17:07.001120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.001086 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-825cw"] Nov 04 03:17:07.067583 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.067543 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 03:17:07.073700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.073668 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-zhmt8"] Nov 04 03:17:07.161531 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.161496 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" path="/var/lib/kubelet/pods/77b0097c-5696-43a3-a6cd-74ff8d85897f/volumes" Nov 04 03:17:07.161966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.161952 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" path="/var/lib/kubelet/pods/e1afcda6-7342-4238-a55b-14b7aee67b61/volumes" Nov 04 03:17:07.188581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.188540 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 03:17:07.192081 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:07.192054 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xr2tj"] Nov 04 03:17:09.160721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:09.160686 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" path="/var/lib/kubelet/pods/8040dd82-b66b-4c36-ad01-c466a63ffc11/volumes" Nov 04 03:17:09.529117 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:09.529078 4784 patch_prober.go:29] interesting pod/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:17:09.529323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:09.529148 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:09.977950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:09.977921 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:17:10.054929 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.054896 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\") pod \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " Nov 04 03:17:10.055141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.055015 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\" (UniqueName: \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\") pod \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\" (UID: \"635ee2a1-40eb-4e16-9738-3b7eaa7d321f\") " Nov 04 03:17:10.055326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.055303 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume" pod "635ee2a1-40eb-4e16-9738-3b7eaa7d321f" (UID: "635ee2a1-40eb-4e16-9738-3b7eaa7d321f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:10.057541 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.057514 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn" pod "635ee2a1-40eb-4e16-9738-3b7eaa7d321f" (UID: "635ee2a1-40eb-4e16-9738-3b7eaa7d321f"). InnerVolumeSpecName "kube-api-access-spwmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:10.156653 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.156569 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spwmn\" (UniqueName: \"kubernetes.io/projected/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-kube-api-access-spwmn\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.156653 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.156600 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635ee2a1-40eb-4e16-9738-3b7eaa7d321f-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.752727 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.752699 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:17:10.816275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.816241 4784 generic.go:356] "Generic (PLEG): container finished" podID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerID="62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be" exitCode=0 Nov 04 03:17:10.816462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.816302 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" Nov 04 03:17:10.816462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.816331 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" event={"ID":"635ee2a1-40eb-4e16-9738-3b7eaa7d321f","Type":"ContainerDied","Data":"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be"} Nov 04 03:17:10.816462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.816392 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc" event={"ID":"635ee2a1-40eb-4e16-9738-3b7eaa7d321f","Type":"ContainerDied","Data":"2413e2e3c4cd0439f76b265fb5a57b331c7f813734bac81478a8c25c0c0027d6"} Nov 04 03:17:10.816462 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.816417 4784 scope.go:122] "RemoveContainer" containerID="62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be" Nov 04 03:17:10.818442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.818417 4784 generic.go:356] "Generic (PLEG): container finished" podID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerID="353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531" exitCode=0 Nov 04 03:17:10.818571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.818475 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" Nov 04 03:17:10.818571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.818499 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerDied","Data":"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531"} Nov 04 03:17:10.818571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.818541 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq" event={"ID":"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e","Type":"ContainerDied","Data":"71f78f8239234f929627d3aac05739a4aad3e39dad01c9eaa7d364f56c0a55f3"} Nov 04 03:17:10.840210 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.840022 4784 scope.go:122] "RemoveContainer" containerID="62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be" Nov 04 03:17:10.840568 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:10.840538 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be\": container with ID starting with 62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be not found: ID does not exist" containerID="62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be" Nov 04 03:17:10.840688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.840579 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be"} err="failed to get container status \"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be\": rpc error: code = NotFound desc = could not find container \"62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be\": container with ID starting with 62a2226898b0f1af70a1c539eb487f4671e94700d1eebe15b16b8ef514b4e6be not found: ID does not exist" Nov 04 03:17:10.840688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.840606 4784 scope.go:122] "RemoveContainer" containerID="353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531" Nov 04 03:17:10.850327 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.850309 4784 scope.go:122] "RemoveContainer" containerID="1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4" Nov 04 03:17:10.859609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.859586 4784 scope.go:122] "RemoveContainer" containerID="353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531" Nov 04 03:17:10.859888 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:10.859869 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531\": container with ID starting with 353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531 not found: ID does not exist" containerID="353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531" Nov 04 03:17:10.859926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.859897 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531"} err="failed to get container status \"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531\": rpc error: code = NotFound desc = could not find container \"353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531\": container with ID starting with 353f47283b4182f3821bbcd7b06f475dcdb1db415b8ea6c259b649c652a9a531 not found: ID does not exist" Nov 04 03:17:10.859926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.859915 4784 scope.go:122] "RemoveContainer" containerID="1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4" Nov 04 03:17:10.860136 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:10.860121 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4\": container with ID starting with 1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4 not found: ID does not exist" containerID="1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4" Nov 04 03:17:10.860171 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.860144 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4"} err="failed to get container status \"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4\": rpc error: code = NotFound desc = could not find container \"1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4\": container with ID starting with 1ab9fa70e7a520ffb29299f1a32429dcb3b47a3780b7d8b5114ac421f1b25aa4 not found: ID does not exist" Nov 04 03:17:10.864464 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864444 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864532 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864510 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864557 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864632 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864621 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864658 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864697 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864801 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864740 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864801 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864764 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:10.864801 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864787 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864832 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\") pod \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\" (UID: \"62ae9b7b-7b5e-44ec-a12b-8171aa02a00e\") " Nov 04 03:17:10.864967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864871 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:10.864967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.864942 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:10.865452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.865295 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.865452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.865327 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.865452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.865342 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.865672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.865483 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:10.865672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.865561 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:10.866888 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.866863 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:10.866986 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.866902 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "kube-api-access-2n5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:10.866986 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.866922 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:10.867077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.867067 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token" pod "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" (UID: "62ae9b7b-7b5e-44ec-a12b-8171aa02a00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:10.966114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966072 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.966114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966108 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.966114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966122 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.966408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966134 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.966408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966145 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:10.966408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:10.966157 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5lf\" (UniqueName: \"kubernetes.io/projected/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e-kube-api-access-2n5lf\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:11.045059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:11.044951 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:17:11.049334 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:11.049304 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db5ef5-5b75965f5-j72vc"] Nov 04 03:17:11.160470 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:11.160438 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" path="/var/lib/kubelet/pods/635ee2a1-40eb-4e16-9738-3b7eaa7d321f/volumes" Nov 04 03:17:11.243080 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:11.243040 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:17:11.247581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:11.247546 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-db5ef5-openshift-default-596b67fbf7-qrgxq"] Nov 04 03:17:13.159990 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:13.159954 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" path="/var/lib/kubelet/pods/62ae9b7b-7b5e-44ec-a12b-8171aa02a00e/volumes" Nov 04 03:17:28.182218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182154 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk"] Nov 04 03:17:28.182759 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182743 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" containerName="revoke-keys" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182764 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" containerName="revoke-keys" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182782 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182787 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182795 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182801 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.182808 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182807 4784 cpu_manager.go:414] "containerMap: removing container" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182811 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182823 4784 cpu_manager.go:414] "containerMap: removing container" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182829 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182839 4784 cpu_manager.go:414] "containerMap: removing container" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182848 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182861 4784 cpu_manager.go:414] "containerMap: removing container" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182865 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182876 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182880 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182888 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182902 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182914 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9d2eadc5-f33e-4517-831d-adfd898c068e" containerName="payload-processing" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182919 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2eadc5-f33e-4517-831d-adfd898c068e" containerName="payload-processing" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182936 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.183006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.182944 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183018 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183026 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183031 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183039 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" containerName="revoke-keys" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183044 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183049 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1afcda6-7342-4238-a55b-14b7aee67b61" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183054 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="80054ecc-72b0-4405-8f2c-2c8c283bc3f3" containerName="discovery" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183061 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183066 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183072 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="635ee2a1-40eb-4e16-9738-3b7eaa7d321f" containerName="payload-pre-processing" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183079 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8040dd82-b66b-4c36-ad01-c466a63ffc11" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183089 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0718ab-45bb-49ce-8d9f-216970d57496" containerName="istio-proxy" Nov 04 03:17:28.183518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.183100 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2eadc5-f33e-4517-831d-adfd898c068e" containerName="payload-processing" Nov 04 03:17:28.188162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.188141 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.248897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.248826 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk"] Nov 04 03:17:28.249330 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.249305 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.249555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.249533 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2wc\" (UniqueName: \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.249633 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.249568 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.295565 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.295523 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:17:28.295565 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.295547 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:17:28.295857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.295578 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:17:28.295857 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.295619 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:17:28.350043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.350003 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.350242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.350084 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2wc\" (UniqueName: \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.350242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.350105 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.350776 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.350748 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.352762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.352733 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.412851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.412818 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2wc\" (UniqueName: \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\") pod \"maas-api-revoke-sub-c27afeac-n5xrk\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.502255 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.502164 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:28.881650 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.881624 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk"] Nov 04 03:17:28.882339 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:17:28.882313 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54257295_d6fe_4960_8a6b_24b596c82f72.slice/crio-53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef WatchSource:0}: Error finding container 53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef: Status 404 returned error can't find the container with id 53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef Nov 04 03:17:28.943598 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:28.943560 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" event={"ID":"54257295-d6fe-4960-8a6b-24b596c82f72","Type":"ContainerStarted","Data":"53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef"} Nov 04 03:17:29.953099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:29.953062 4784 generic.go:356] "Generic (PLEG): container finished" podID="54257295-d6fe-4960-8a6b-24b596c82f72" containerID="4e8ef6f21999cf1ee45049f01bc508863d3b504afb4981d4b696fd6c837f2221" exitCode=0 Nov 04 03:17:29.953557 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:29.953152 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" event={"ID":"54257295-d6fe-4960-8a6b-24b596c82f72","Type":"ContainerDied","Data":"4e8ef6f21999cf1ee45049f01bc508863d3b504afb4981d4b696fd6c837f2221"} Nov 04 03:17:31.112721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.112692 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:31.180830 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.180794 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\") pod \"54257295-d6fe-4960-8a6b-24b596c82f72\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " Nov 04 03:17:31.181045 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.180875 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\") pod \"54257295-d6fe-4960-8a6b-24b596c82f72\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " Nov 04 03:17:31.181045 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.180931 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\" (UniqueName: \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\") pod \"54257295-d6fe-4960-8a6b-24b596c82f72\" (UID: \"54257295-d6fe-4960-8a6b-24b596c82f72\") " Nov 04 03:17:31.181259 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.181182 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca" pod "54257295-d6fe-4960-8a6b-24b596c82f72" (UID: "54257295-d6fe-4960-8a6b-24b596c82f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:31.181420 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.181406 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:31.183254 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.183229 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc" pod "54257295-d6fe-4960-8a6b-24b596c82f72" (UID: "54257295-d6fe-4960-8a6b-24b596c82f72"). InnerVolumeSpecName "kube-api-access-6k2wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:31.183360 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.183294 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca" pod "54257295-d6fe-4960-8a6b-24b596c82f72" (UID: "54257295-d6fe-4960-8a6b-24b596c82f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:31.282677 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.282576 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54257295-d6fe-4960-8a6b-24b596c82f72-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:31.282677 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.282622 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2wc\" (UniqueName: \"kubernetes.io/projected/54257295-d6fe-4960-8a6b-24b596c82f72-kube-api-access-6k2wc\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:31.966404 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.966368 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" Nov 04 03:17:31.966404 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.966395 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk" event={"ID":"54257295-d6fe-4960-8a6b-24b596c82f72","Type":"ContainerDied","Data":"53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef"} Nov 04 03:17:31.966652 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:31.966443 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53678a70535b802a6153ed51d5ae6dbe70e880700cd1623b6fc3ae9894d596ef" Nov 04 03:17:32.485053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:32.485012 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 03:17:33.847701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:33.847669 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:17:33.853119 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:33.853095 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:17:33.853870 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:33.853850 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:17:33.858924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:33.858907 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:17:36.403024 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:36.402978 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:17:36.599085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.599052 4784 scope.go:122] "RemoveContainer" containerID="a3dc056bea0932df588fad807083b4b3b120070c9044d2f5a1e0df317fd3d655" Nov 04 03:17:36.612979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.612951 4784 scope.go:122] "RemoveContainer" containerID="5642e393584bf12901e2072bbbe4131e677c6558731d400092e41a1d505f832c" Nov 04 03:17:36.624738 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.624701 4784 scope.go:122] "RemoveContainer" containerID="ebbde593ef0ae571fdbc673f5ed09e12bc3022d2e656eecb08c49784620fcf71" Nov 04 03:17:36.634743 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.634720 4784 scope.go:122] "RemoveContainer" containerID="a0d888cf956dd9d279d41838ce3ed02c8bbb423a197b44d6e605de85430d8589" Nov 04 03:17:36.644419 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.644395 4784 scope.go:122] "RemoveContainer" containerID="1302c3f71c3cf57ba263e529542bc39cd6b5df315186ab64cb2527fa9ef6a38d" Nov 04 03:17:36.653486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.653465 4784 scope.go:122] "RemoveContainer" containerID="b2b7bab0ed68ebfcdbc1c1109dc7858c61332611cd5ce674d5e72ccbec85ca1d" Nov 04 03:17:36.662712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.662692 4784 scope.go:122] "RemoveContainer" containerID="1feb37c619adbe866b67923ab8f16a21550597ce66a5a1529e2f44527c3fccdf" Nov 04 03:17:36.672216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:36.672170 4784 scope.go:122] "RemoveContainer" containerID="8c60c6770cc447563756bbe7ed9be155c729dad3dd8e85ae3a97bce2e59b7284" Nov 04 03:17:44.837979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:44.837943 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:17:44.838553 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:44.838170 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" containerID="cri-o://dc955c4c77f306b444048e24b93fc11802c38893fb727abeb460368d26409ad9" gracePeriod=30 Nov 04 03:17:45.000819 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.000781 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm"] Nov 04 03:17:45.001308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001295 4784 cpu_manager.go:414] "containerMap: removing container" podUID="54257295-d6fe-4960-8a6b-24b596c82f72" containerName="revoke-keys" Nov 04 03:17:45.001363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001310 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54257295-d6fe-4960-8a6b-24b596c82f72" containerName="revoke-keys" Nov 04 03:17:45.001363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001327 4784 cpu_manager.go:414] "containerMap: removing container" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:45.001363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001333 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ae9b7b-7b5e-44ec-a12b-8171aa02a00e" containerName="istio-proxy" Nov 04 03:17:45.001363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001341 4784 cpu_manager.go:414] "containerMap: removing container" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:45.001363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001348 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b0097c-5696-43a3-a6cd-74ff8d85897f" containerName="istio-proxy" Nov 04 03:17:45.001532 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.001419 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="54257295-d6fe-4960-8a6b-24b596c82f72" containerName="revoke-keys" Nov 04 03:17:45.006811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.006794 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.061002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.060800 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:17:45.061002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.060871 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:17:45.061002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.060924 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:17:45.061002 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.060987 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:17:45.062508 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.062481 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm"] Nov 04 03:17:45.137950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.137849 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.137950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.137911 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.138236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.138062 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcfj\" (UniqueName: \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.239148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.239106 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.239148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.239151 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.239417 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.239352 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcfj\" (UniqueName: \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.240089 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.240062 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.241769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.241746 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.301894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.301861 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcfj\" (UniqueName: \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\") pod \"maas-api-revoke-sub-a9d38f41-wmkcm\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.320668 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.320631 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:45.590592 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.590559 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 03:17:45.716950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:45.716906 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm"] Nov 04 03:17:45.719299 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:17:45.719248 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d2e8d1_1d68_45ae_9c7a_426a6ff90b70.slice/crio-b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6 WatchSource:0}: Error finding container b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6: Status 404 returned error can't find the container with id b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6 Nov 04 03:17:46.056121 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:46.056070 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" event={"ID":"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70","Type":"ContainerStarted","Data":"b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6"} Nov 04 03:17:47.074249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.074211 4784 generic.go:356] "Generic (PLEG): container finished" podID="69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" containerID="1bbac6e3473ac9eba7aa33b4433200762a897cff71b236abd1afb08027e0e519" exitCode=0 Nov 04 03:17:47.074770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.074292 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" event={"ID":"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70","Type":"ContainerDied","Data":"1bbac6e3473ac9eba7aa33b4433200762a897cff71b236abd1afb08027e0e519"} Nov 04 03:17:47.507250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.507207 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" containerID="cri-o://54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f" gracePeriod=45 Nov 04 03:17:47.775626 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.775592 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 03:17:47.867528 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867489 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.867745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867603 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.867745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867643 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\" (UniqueName: \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.867745 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867682 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.868116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867778 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\" (UniqueName: \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.868116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867835 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\") pod \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\" (UID: \"d83d1144-9af1-4bf3-9a77-dab59d43cc30\") " Nov 04 03:17:47.868116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.867852 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.868320 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.868302 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.868503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.868468 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.869048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.869026 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.871388 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.871344 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.871855 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.871396 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "kube-api-access-xgh5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:47.871855 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.871496 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs" pod "d83d1144-9af1-4bf3-9a77-dab59d43cc30" (UID: "d83d1144-9af1-4bf3-9a77-dab59d43cc30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:47.968960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.968925 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.968960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.968958 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83d1144-9af1-4bf3-9a77-dab59d43cc30-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.968960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.968967 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.969238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.968978 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgh5x\" (UniqueName: \"kubernetes.io/projected/d83d1144-9af1-4bf3-9a77-dab59d43cc30-kube-api-access-xgh5x\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.969238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:47.968989 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83d1144-9af1-4bf3-9a77-dab59d43cc30-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.082399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.082295 4784 generic.go:356] "Generic (PLEG): container finished" podID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerID="54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f" exitCode=0 Nov 04 03:17:48.082399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.082336 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" Nov 04 03:17:48.082399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.082363 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" event={"ID":"d83d1144-9af1-4bf3-9a77-dab59d43cc30","Type":"ContainerDied","Data":"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f"} Nov 04 03:17:48.082957 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.082410 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" event={"ID":"d83d1144-9af1-4bf3-9a77-dab59d43cc30","Type":"ContainerDied","Data":"506559081b343190265850b2066dc4c380d85441804c5af3507af69ce2cef6ee"} Nov 04 03:17:48.082957 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.082431 4784 scope.go:122] "RemoveContainer" containerID="54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f" Nov 04 03:17:48.100535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.100505 4784 scope.go:122] "RemoveContainer" containerID="54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f" Nov 04 03:17:48.100991 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:48.100963 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f\": container with ID starting with 54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f not found: ID does not exist" containerID="54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f" Nov 04 03:17:48.101120 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.101004 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f"} err="failed to get container status \"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f\": rpc error: code = NotFound desc = could not find container \"54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f\": container with ID starting with 54a456795b7d21f7bdbae54e3d2518a9240ce36dc136034ce381d4d1da1ba78f not found: ID does not exist" Nov 04 03:17:48.217712 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.217677 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:48.251799 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.251759 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 03:17:48.254987 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.254965 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9"] Nov 04 03:17:48.373275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.373168 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\") pod \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " Nov 04 03:17:48.373275 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.373261 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\" (UniqueName: \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\") pod \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " Nov 04 03:17:48.373473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.373367 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\") pod \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\" (UID: \"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70\") " Nov 04 03:17:48.373614 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.373591 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca" pod "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" (UID: "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:48.373880 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.373857 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.375662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.375634 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca" pod "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" (UID: "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:48.375960 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.375945 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj" pod "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" (UID: "69d2e8d1-1d68-45ae-9c7a-426a6ff90b70"). InnerVolumeSpecName "kube-api-access-bfcfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:48.474906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.474865 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcfj\" (UniqueName: \"kubernetes.io/projected/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-kube-api-access-bfcfj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.474906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.474899 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.718726 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:48.717576 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/simulator-w2-a0bcbc-kserve-68759db55d-zlpr9" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 03:17:49.091147 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:49.091062 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" event={"ID":"69d2e8d1-1d68-45ae-9c7a-426a6ff90b70","Type":"ContainerDied","Data":"b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6"} Nov 04 03:17:49.091147 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:49.091093 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm" Nov 04 03:17:49.091147 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:49.091112 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e9bbff07457fe345c94bec891f1ed4972600760fd63f4b8283417e4ec512f6" Nov 04 03:17:49.161128 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:49.161096 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" path="/var/lib/kubelet/pods/d83d1144-9af1-4bf3-9a77-dab59d43cc30/volumes" Nov 04 03:17:50.101860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.101790 4784 generic.go:356] "Generic (PLEG): container finished" podID="f7b4b264-53bb-429c-8215-59856f253f65" containerID="dc955c4c77f306b444048e24b93fc11802c38893fb727abeb460368d26409ad9" exitCode=0 Nov 04 03:17:50.102351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.101859 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerDied","Data":"dc955c4c77f306b444048e24b93fc11802c38893fb727abeb460368d26409ad9"} Nov 04 03:17:50.102351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.101909 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" event={"ID":"f7b4b264-53bb-429c-8215-59856f253f65","Type":"ContainerDied","Data":"c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd"} Nov 04 03:17:50.102351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.101926 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6fb3d97c6bd3be1b52b70c9a16d3674a158d154922fd992a47313d1bf5510bd" Nov 04 03:17:50.102351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.101946 4784 scope.go:122] "RemoveContainer" containerID="834cbb280dadc117d66c7e2158e4693953909d032a01f027100198c518c8c6ed" Nov 04 03:17:50.105169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.105148 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:17:50.193287 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193253 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193287 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193300 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193325 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193351 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193397 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193547 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193460 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193548 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193586 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193651 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\") pod \"f7b4b264-53bb-429c-8215-59856f253f65\" (UID: \"f7b4b264-53bb-429c-8215-59856f253f65\") " Nov 04 03:17:50.193769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193660 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.193769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193694 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.194023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193818 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.194023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193835 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.194023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.193742 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:50.194181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.194052 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b4b264-53bb-429c-8215-59856f253f65-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.194181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.194073 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.194181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.194088 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.194181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.194102 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.194181 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.194117 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.195964 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.195937 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:50.196116 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.195982 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "kube-api-access-78922". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:50.196409 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.196384 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:50.196549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.196444 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy" pod "f7b4b264-53bb-429c-8215-59856f253f65" (UID: "f7b4b264-53bb-429c-8215-59856f253f65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.294754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.294719 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b4b264-53bb-429c-8215-59856f253f65-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.294754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.294748 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78922\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-kube-api-access-78922\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.294754 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.294758 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b4b264-53bb-429c-8215-59856f253f65-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.295086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.294768 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b4b264-53bb-429c-8215-59856f253f65-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.591133 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.591060 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" podUID="88e41c29-9039-4264-aee1-3e6b7d27444a" containerName="payload-processing" containerID="cri-o://435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c" gracePeriod=25 Nov 04 03:17:50.831804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:50.831779 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 03:17:51.002818 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.002783 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\") pod \"88e41c29-9039-4264-aee1-3e6b7d27444a\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " Nov 04 03:17:51.003016 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.002828 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\") pod \"88e41c29-9039-4264-aee1-3e6b7d27444a\" (UID: \"88e41c29-9039-4264-aee1-3e6b7d27444a\") " Nov 04 03:17:51.003172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.003150 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume" pod "88e41c29-9039-4264-aee1-3e6b7d27444a" (UID: "88e41c29-9039-4264-aee1-3e6b7d27444a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:51.005085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.005065 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l" pod "88e41c29-9039-4264-aee1-3e6b7d27444a" (UID: "88e41c29-9039-4264-aee1-3e6b7d27444a"). InnerVolumeSpecName "kube-api-access-tqn5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:51.104551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.104513 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e41c29-9039-4264-aee1-3e6b7d27444a-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:51.104551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.104547 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/88e41c29-9039-4264-aee1-3e6b7d27444a-kube-api-access-tqn5l\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:51.109497 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.109470 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr" Nov 04 03:17:51.110883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.110856 4784 generic.go:356] "Generic (PLEG): container finished" podID="88e41c29-9039-4264-aee1-3e6b7d27444a" containerID="435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c" exitCode=0 Nov 04 03:17:51.111012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.110910 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" Nov 04 03:17:51.111012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.110916 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" event={"ID":"88e41c29-9039-4264-aee1-3e6b7d27444a","Type":"ContainerDied","Data":"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c"} Nov 04 03:17:51.111143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.111064 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9" event={"ID":"88e41c29-9039-4264-aee1-3e6b7d27444a","Type":"ContainerDied","Data":"7d9c136fdc23151faf2608177474173d9c7b1475c536bb426ec6446fc9693312"} Nov 04 03:17:51.111143 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.111099 4784 scope.go:122] "RemoveContainer" containerID="435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c" Nov 04 03:17:51.122283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.122264 4784 scope.go:122] "RemoveContainer" containerID="435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c" Nov 04 03:17:51.122658 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:17:51.122634 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c\": container with ID starting with 435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c not found: ID does not exist" containerID="435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c" Nov 04 03:17:51.122737 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.122670 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c"} err="failed to get container status \"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c\": rpc error: code = NotFound desc = could not find container \"435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c\": container with ID starting with 435306c618233467811dcbde6e67dc8c887b93c3291e477d31250b76ceaeae7c not found: ID does not exist" Nov 04 03:17:51.347747 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.347716 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 03:17:51.353024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.352996 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a0bcbc-5bf4697664-l2vk9"] Nov 04 03:17:51.464818 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.464776 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:17:51.469069 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:51.469042 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-095081-openshift-default-59dfcb6dcf-vf8xr"] Nov 04 03:17:53.160342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:53.160302 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e41c29-9039-4264-aee1-3e6b7d27444a" path="/var/lib/kubelet/pods/88e41c29-9039-4264-aee1-3e6b7d27444a/volumes" Nov 04 03:17:53.160952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:17:53.160931 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b4b264-53bb-429c-8215-59856f253f65" path="/var/lib/kubelet/pods/f7b4b264-53bb-429c-8215-59856f253f65/volumes" Nov 04 03:18:06.400265 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:06.400227 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:18:12.539681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:12.539597 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 03:18:12.540087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:12.539821 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" containerID="cri-o://3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b" gracePeriod=30 Nov 04 03:18:17.797316 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.797290 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 03:18:17.875901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.875812 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.875923 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.875986 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876018 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876061 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876114 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876163 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876255 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876312 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\") pod \"95642dea-64ed-401d-893f-d1def7ecee3e\" (UID: \"95642dea-64ed-401d-893f-d1def7ecee3e\") " Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876317 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.876355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876332 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.876665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876564 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.876868 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876837 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.876973 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.876901 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:17.877645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.877619 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.877768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.877648 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95642dea-64ed-401d-893f-d1def7ecee3e-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.877768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.877663 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.877768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.877677 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.877768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.877689 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.878878 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.878840 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:18:17.878878 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.878861 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.879032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.878958 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "kube-api-access-qrrx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:17.879158 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.879141 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token" pod "95642dea-64ed-401d-893f-d1def7ecee3e" (UID: "95642dea-64ed-401d-893f-d1def7ecee3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:17.978862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.978816 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.978862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.978856 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrx7\" (UniqueName: \"kubernetes.io/projected/95642dea-64ed-401d-893f-d1def7ecee3e-kube-api-access-qrrx7\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.978862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.978866 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95642dea-64ed-401d-893f-d1def7ecee3e-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.979100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:17.978875 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95642dea-64ed-401d-893f-d1def7ecee3e-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:18.279082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.279044 4784 generic.go:356] "Generic (PLEG): container finished" podID="95642dea-64ed-401d-893f-d1def7ecee3e" containerID="3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b" exitCode=0 Nov 04 03:18:18.279308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.279097 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" Nov 04 03:18:18.279308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.279134 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" event={"ID":"95642dea-64ed-401d-893f-d1def7ecee3e","Type":"ContainerDied","Data":"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b"} Nov 04 03:18:18.279308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.279209 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9" event={"ID":"95642dea-64ed-401d-893f-d1def7ecee3e","Type":"ContainerDied","Data":"a51babf2ecc13b45d833e2462bc5addfcf8a8fd3c41a93b5862f1223ed85d71a"} Nov 04 03:18:18.279308 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.279243 4784 scope.go:122] "RemoveContainer" containerID="3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b" Nov 04 03:18:18.290701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.290683 4784 scope.go:122] "RemoveContainer" containerID="3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b" Nov 04 03:18:18.290969 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:18.290950 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b\": container with ID starting with 3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b not found: ID does not exist" containerID="3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b" Nov 04 03:18:18.291054 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.290979 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b"} err="failed to get container status \"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b\": rpc error: code = NotFound desc = could not find container \"3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b\": container with ID starting with 3493b4679af85718e41f2d6cc5747a7622b4ad247693094b24c1d3676a45668b not found: ID does not exist" Nov 04 03:18:18.448646 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.448609 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 03:18:18.456608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:18.456571 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-bjdq9"] Nov 04 03:18:19.159639 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:19.159603 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" path="/var/lib/kubelet/pods/95642dea-64ed-401d-893f-d1def7ecee3e/volumes" Nov 04 03:18:28.368615 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.368580 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42"] Nov 04 03:18:28.369026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369010 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:28.369026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369020 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369030 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369036 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369060 4784 cpu_manager.go:414] "containerMap: removing container" podUID="88e41c29-9039-4264-aee1-3e6b7d27444a" containerName="payload-processing" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369064 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e41c29-9039-4264-aee1-3e6b7d27444a" containerName="payload-processing" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369074 4784 cpu_manager.go:414] "containerMap: removing container" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369079 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369097 4784 cpu_manager.go:414] "containerMap: removing container" podUID="69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" containerName="revoke-keys" Nov 04 03:18:28.369100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369101 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" containerName="revoke-keys" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369164 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e41c29-9039-4264-aee1-3e6b7d27444a" containerName="payload-processing" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369172 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="95642dea-64ed-401d-893f-d1def7ecee3e" containerName="istio-proxy" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369183 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369225 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" containerName="revoke-keys" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369233 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:28.369385 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.369285 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83d1144-9af1-4bf3-9a77-dab59d43cc30" containerName="main" Nov 04 03:18:28.373915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.373897 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.431057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.431019 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42"] Nov 04 03:18:28.478346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.478313 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:18:28.478346 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.478315 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:18:28.478569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.478360 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:18:28.478569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.478419 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:18:28.490279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.490255 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.490436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.490381 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.490500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.490444 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765nq\" (UniqueName: \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.590974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.590937 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.591172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.591015 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765nq\" (UniqueName: \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.591172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.591080 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.591876 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.591851 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.593546 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.593526 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.655149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.655075 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765nq\" (UniqueName: \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\") pod \"maas-api-revoke-sub-3e54fb7e-xsm42\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:28.687018 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:28.686985 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:29.068416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:29.068386 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42"] Nov 04 03:18:29.069537 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:18:29.069512 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9a4ea3_d64d_4d49_a949_34e1b9d61d81.slice/crio-984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93 WatchSource:0}: Error finding container 984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93: Status 404 returned error can't find the container with id 984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93 Nov 04 03:18:29.349889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:29.349803 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" event={"ID":"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81","Type":"ContainerStarted","Data":"984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93"} Nov 04 03:18:30.358730 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:30.358644 4784 generic.go:356] "Generic (PLEG): container finished" podID="dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" containerID="c68058f04b3583538881a3da5473dfbe856e180e0859f78d4c95db74333508e3" exitCode=0 Nov 04 03:18:30.359107 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:30.358725 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" event={"ID":"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81","Type":"ContainerDied","Data":"c68058f04b3583538881a3da5473dfbe856e180e0859f78d4c95db74333508e3"} Nov 04 03:18:31.510908 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.510876 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:31.623309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.623218 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\" (UniqueName: \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\") pod \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " Nov 04 03:18:31.623535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.623392 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\") pod \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " Nov 04 03:18:31.623535 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.623447 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\") pod \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\" (UID: \"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81\") " Nov 04 03:18:31.623781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.623739 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca" pod "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" (UID: "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:31.623907 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.623856 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:31.625605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.625585 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca" pod "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" (UID: "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:31.625673 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.625621 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq" pod "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" (UID: "dc9a4ea3-d64d-4d49-a949-34e1b9d61d81"). InnerVolumeSpecName "kube-api-access-765nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:31.724600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.724567 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765nq\" (UniqueName: \"kubernetes.io/projected/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-kube-api-access-765nq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:31.724600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:31.724596 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.374511 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:32.374476 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" Nov 04 03:18:32.374704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:32.374506 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42" event={"ID":"dc9a4ea3-d64d-4d49-a949-34e1b9d61d81","Type":"ContainerDied","Data":"984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93"} Nov 04 03:18:32.374704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:32.374578 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984d855986ac13800f40c1099c6147c49ea090f10be9431fb87b386413be3b93" Nov 04 03:18:32.709384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:32.709351 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 03:18:32.714536 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:32.714510 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 03:18:36.399834 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:36.399797 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:18:41.513912 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.513874 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8"] Nov 04 03:18:41.514366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.514353 4784 cpu_manager.go:414] "containerMap: removing container" podUID="dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" containerName="revoke-keys" Nov 04 03:18:41.514412 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.514369 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" containerName="revoke-keys" Nov 04 03:18:41.514448 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.514416 4784 cpu_manager.go:414] "containerMap: removing container" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:41.514448 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.514422 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b4b264-53bb-429c-8215-59856f253f65" containerName="istio-proxy" Nov 04 03:18:41.514518 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.514505 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" containerName="revoke-keys" Nov 04 03:18:41.519058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.519042 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.575070 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.575033 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8"] Nov 04 03:18:41.622516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.622476 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:18:41.622700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.622524 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:18:41.622749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.622708 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:18:41.623573 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.623555 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:18:41.626051 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.626023 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.626242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.626225 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nghg\" (UniqueName: \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.626359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.626343 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.727303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.727259 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nghg\" (UniqueName: \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.727514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.727359 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.727514 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.727433 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.728034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.728012 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.730112 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.730090 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.790146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.790062 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nghg\" (UniqueName: \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\") pod \"maas-api-revoke-sub-4c0eb77b-2h4p8\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:41.833024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:41.832990 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:42.052874 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:42.052775 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 03:18:42.161641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:42.161608 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 03:18:42.209759 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:18:42.209726 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ba441a_2429_4d63_b622_9bc9965c524d.slice/crio-4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888 WatchSource:0}: Error finding container 4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888: Status 404 returned error can't find the container with id 4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888 Nov 04 03:18:42.212348 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:42.212325 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8"] Nov 04 03:18:42.437098 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:42.437063 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" event={"ID":"28ba441a-2429-4d63-b622-9bc9965c524d","Type":"ContainerStarted","Data":"4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888"} Nov 04 03:18:43.444703 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:43.444663 4784 generic.go:356] "Generic (PLEG): container finished" podID="28ba441a-2429-4d63-b622-9bc9965c524d" containerID="dff75617fc264226eabe6f8a8811f708fd2ec9ef2cef68c30b96a6ed3ac48bce" exitCode=0 Nov 04 03:18:43.445108 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:43.444734 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" event={"ID":"28ba441a-2429-4d63-b622-9bc9965c524d","Type":"ContainerDied","Data":"dff75617fc264226eabe6f8a8811f708fd2ec9ef2cef68c30b96a6ed3ac48bce"} Nov 04 03:18:44.585577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.585552 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:44.657975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.657946 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\") pod \"28ba441a-2429-4d63-b622-9bc9965c524d\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " Nov 04 03:18:44.658182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.657994 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\" (UniqueName: \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\") pod \"28ba441a-2429-4d63-b622-9bc9965c524d\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " Nov 04 03:18:44.658182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.658169 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\") pod \"28ba441a-2429-4d63-b622-9bc9965c524d\" (UID: \"28ba441a-2429-4d63-b622-9bc9965c524d\") " Nov 04 03:18:44.658530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.658503 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca" pod "28ba441a-2429-4d63-b622-9bc9965c524d" (UID: "28ba441a-2429-4d63-b622-9bc9965c524d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:44.660603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.660574 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca" pod "28ba441a-2429-4d63-b622-9bc9965c524d" (UID: "28ba441a-2429-4d63-b622-9bc9965c524d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:44.660688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.660604 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg" pod "28ba441a-2429-4d63-b622-9bc9965c524d" (UID: "28ba441a-2429-4d63-b622-9bc9965c524d"). InnerVolumeSpecName "kube-api-access-9nghg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:44.760105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.760025 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:44.760105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.760055 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba441a-2429-4d63-b622-9bc9965c524d-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:44.760105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:44.760065 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nghg\" (UniqueName: \"kubernetes.io/projected/28ba441a-2429-4d63-b622-9bc9965c524d-kube-api-access-9nghg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:45.458477 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:45.458391 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" Nov 04 03:18:45.458642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:45.458387 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8" event={"ID":"28ba441a-2429-4d63-b622-9bc9965c524d","Type":"ContainerDied","Data":"4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888"} Nov 04 03:18:45.458642 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:45.458520 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b531d78249567e2405b120cdbcd40cb72877071fe136c0b4808a4622d50f888" Nov 04 03:18:47.053453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.053404 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" podUID="02459976-aea1-488d-83f7-f2525df146ec" containerName="payload-processing" containerID="cri-o://31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad" gracePeriod=25 Nov 04 03:18:47.169735 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.161976 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerName="payload-pre-processing" containerID="cri-o://0b50b0474742bfab43528e56f5f76dbd304ac021aba120c03c015e85898646e7" gracePeriod=25 Nov 04 03:18:47.200211 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:18:47.199364 4784 logging.go:55] [core] [Channel #6923 SubChannel #6924]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.70:50052", ServerName: "10.133.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.70:50052: connect: connection refused" Nov 04 03:18:47.417234 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.415870 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 03:18:47.474591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.474554 4784 generic.go:356] "Generic (PLEG): container finished" podID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerID="0b50b0474742bfab43528e56f5f76dbd304ac021aba120c03c015e85898646e7" exitCode=0 Nov 04 03:18:47.474707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.474673 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" event={"ID":"7ab74e6d-4c65-42aa-99d1-42e8146a943f","Type":"ContainerDied","Data":"0b50b0474742bfab43528e56f5f76dbd304ac021aba120c03c015e85898646e7"} Nov 04 03:18:47.478628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.478296 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" event={"ID":"7ab74e6d-4c65-42aa-99d1-42e8146a943f","Type":"ContainerDied","Data":"ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d"} Nov 04 03:18:47.478628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.478371 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9139e42ba2c28eead0e3b02b963580b5156adbab07c6f750c950b5747c232d" Nov 04 03:18:47.489858 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.489822 4784 generic.go:356] "Generic (PLEG): container finished" podID="02459976-aea1-488d-83f7-f2525df146ec" containerID="31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad" exitCode=0 Nov 04 03:18:47.490003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.489987 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" event={"ID":"02459976-aea1-488d-83f7-f2525df146ec","Type":"ContainerDied","Data":"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad"} Nov 04 03:18:47.490071 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.490033 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" event={"ID":"02459976-aea1-488d-83f7-f2525df146ec","Type":"ContainerDied","Data":"d8fc1172616130d02a809e5e23585fb4f7219fae5d300c614d2979c41ae0ab45"} Nov 04 03:18:47.490071 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.490059 4784 scope.go:122] "RemoveContainer" containerID="31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad" Nov 04 03:18:47.490228 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.490073 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm" Nov 04 03:18:47.495847 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.495826 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 03:18:47.501598 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.501578 4784 scope.go:122] "RemoveContainer" containerID="31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad" Nov 04 03:18:47.501873 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:47.501854 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad\": container with ID starting with 31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad not found: ID does not exist" containerID="31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad" Nov 04 03:18:47.501932 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.501883 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad"} err="failed to get container status \"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad\": rpc error: code = NotFound desc = could not find container \"31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad\": container with ID starting with 31d0089cad167b7b7f3e02b4e45d8658ab270a23acb430bba46b8ea8602494ad not found: ID does not exist" Nov 04 03:18:47.505458 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.505432 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\" (UniqueName: \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\") pod \"02459976-aea1-488d-83f7-f2525df146ec\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " Nov 04 03:18:47.505551 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.505481 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\") pod \"02459976-aea1-488d-83f7-f2525df146ec\" (UID: \"02459976-aea1-488d-83f7-f2525df146ec\") " Nov 04 03:18:47.505852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.505832 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume" pod "02459976-aea1-488d-83f7-f2525df146ec" (UID: "02459976-aea1-488d-83f7-f2525df146ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:47.507910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.507881 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk" pod "02459976-aea1-488d-83f7-f2525df146ec" (UID: "02459976-aea1-488d-83f7-f2525df146ec"). InnerVolumeSpecName "kube-api-access-p88sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:47.606383 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.606345 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\" (UniqueName: \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\") pod \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " Nov 04 03:18:47.606602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.606525 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\") pod \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\" (UID: \"7ab74e6d-4c65-42aa-99d1-42e8146a943f\") " Nov 04 03:18:47.606869 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.606847 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88sk\" (UniqueName: \"kubernetes.io/projected/02459976-aea1-488d-83f7-f2525df146ec-kube-api-access-p88sk\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:47.606972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.606873 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02459976-aea1-488d-83f7-f2525df146ec-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:47.607033 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.606978 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume" pod "7ab74e6d-4c65-42aa-99d1-42e8146a943f" (UID: "7ab74e6d-4c65-42aa-99d1-42e8146a943f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:47.608873 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.608848 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq" pod "7ab74e6d-4c65-42aa-99d1-42e8146a943f" (UID: "7ab74e6d-4c65-42aa-99d1-42e8146a943f"). InnerVolumeSpecName "kube-api-access-q9qhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:47.708170 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.708119 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qhq\" (UniqueName: \"kubernetes.io/projected/7ab74e6d-4c65-42aa-99d1-42e8146a943f-kube-api-access-q9qhq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:47.708170 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.708169 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab74e6d-4c65-42aa-99d1-42e8146a943f-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:47.731303 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.731225 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" containerID="cri-o://ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442" gracePeriod=45 Nov 04 03:18:47.736066 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.736030 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": dial tcp 10.133.0.121:8000: connect: connection refused" Nov 04 03:18:47.741715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.741681 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" containerID="cri-o://144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1" gracePeriod=45 Nov 04 03:18:47.990233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.987521 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 03:18:47.992162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:47.992127 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-203c35-688854f56b-jk6wm"] Nov 04 03:18:48.004274 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.004242 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 03:18:48.040484 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.040452 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 03:18:48.112506 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112468 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112577 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\" (UniqueName: \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112628 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\" (UniqueName: \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112663 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112750 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.112820 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\") pod \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\" (UID: \"464cc908-f59f-4ce3-9dd1-96a11ebf3188\") " Nov 04 03:18:48.113353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.113272 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.113490 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.113472 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.113602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.113578 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.116289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.116249 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "kube-api-access-98z9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:48.119397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.119358 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.136371 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.136333 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs" pod "464cc908-f59f-4ce3-9dd1-96a11ebf3188" (UID: "464cc908-f59f-4ce3-9dd1-96a11ebf3188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:48.195957 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.195919 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.70:50052\" within 1s: context deadline exceeded" Nov 04 03:18:48.213894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.213857 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.213908 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.213946 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214040 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214091 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\" (UniqueName: \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214116 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\") pod \"1b298ff7-1c44-4582-a93c-c3655e1778cb\" (UID: \"1b298ff7-1c44-4582-a93c-c3655e1778cb\") " Nov 04 03:18:48.214338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214234 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.214338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214250 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.214496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214472 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.214799 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214780 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214800 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214809 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214819 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214827 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214839 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98z9t\" (UniqueName: \"kubernetes.io/projected/464cc908-f59f-4ce3-9dd1-96a11ebf3188-kube-api-access-98z9t\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214853 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464cc908-f59f-4ce3-9dd1-96a11ebf3188-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214864 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.214879 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.214875 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464cc908-f59f-4ce3-9dd1-96a11ebf3188-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.216781 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.216754 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "kube-api-access-nz46j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:48.216900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.216805 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:48.216900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.216817 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm" pod "1b298ff7-1c44-4582-a93c-c3655e1778cb" (UID: "1b298ff7-1c44-4582-a93c-c3655e1778cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.315556 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.315460 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b298ff7-1c44-4582-a93c-c3655e1778cb-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.315556 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.315496 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz46j\" (UniqueName: \"kubernetes.io/projected/1b298ff7-1c44-4582-a93c-c3655e1778cb-kube-api-access-nz46j\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.315556 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.315512 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b298ff7-1c44-4582-a93c-c3655e1778cb-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.497341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.497303 4784 generic.go:356] "Generic (PLEG): container finished" podID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerID="144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1" exitCode=0 Nov 04 03:18:48.497548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.497368 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" Nov 04 03:18:48.497548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.497392 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" event={"ID":"1b298ff7-1c44-4582-a93c-c3655e1778cb","Type":"ContainerDied","Data":"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1"} Nov 04 03:18:48.497548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.497431 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg" event={"ID":"1b298ff7-1c44-4582-a93c-c3655e1778cb","Type":"ContainerDied","Data":"f47ccd28211ea81b64d614fa6de530a1a0bb8eb45f17ee134b9a4ddf5e2efecb"} Nov 04 03:18:48.497548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.497450 4784 scope.go:122] "RemoveContainer" containerID="144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1" Nov 04 03:18:48.498930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.498897 4784 generic.go:356] "Generic (PLEG): container finished" podID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerID="ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442" exitCode=0 Nov 04 03:18:48.499065 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.498940 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" Nov 04 03:18:48.499065 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.498989 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" event={"ID":"464cc908-f59f-4ce3-9dd1-96a11ebf3188","Type":"ContainerDied","Data":"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442"} Nov 04 03:18:48.499065 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.499015 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft" event={"ID":"464cc908-f59f-4ce3-9dd1-96a11ebf3188","Type":"ContainerDied","Data":"8e0357c3c6396d1d190a9eb38570c68bd039c383b74a28e04d15ed9f7fe0775b"} Nov 04 03:18:48.500272 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.500246 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw" Nov 04 03:18:48.509975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.509958 4784 scope.go:122] "RemoveContainer" containerID="144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1" Nov 04 03:18:48.510349 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:48.510323 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1\": container with ID starting with 144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1 not found: ID does not exist" containerID="144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1" Nov 04 03:18:48.510460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.510363 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1"} err="failed to get container status \"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1\": rpc error: code = NotFound desc = could not find container \"144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1\": container with ID starting with 144b58ef184428a4c4d92df59af0e1f71bfd2b45a5f6b5dd4d0b41a47398cfd1 not found: ID does not exist" Nov 04 03:18:48.510460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.510388 4784 scope.go:122] "RemoveContainer" containerID="ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442" Nov 04 03:18:48.520552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.520532 4784 scope.go:122] "RemoveContainer" containerID="ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442" Nov 04 03:18:48.520858 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:18:48.520842 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442\": container with ID starting with ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442 not found: ID does not exist" containerID="ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442" Nov 04 03:18:48.520925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.520869 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442"} err="failed to get container status \"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442\": rpc error: code = NotFound desc = could not find container \"ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442\": container with ID starting with ad398da55aa606169b24f9eaf63a946e19611a41f09239f9bb6c6209382d5442 not found: ID does not exist" Nov 04 03:18:48.666235 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.666203 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 03:18:48.671542 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.671512 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-203c35-5bb8b769bc-q9wmw"] Nov 04 03:18:48.782977 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.782940 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 03:18:48.788829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.788790 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-w4-203c35-kserve-78f575d697-qclvg"] Nov 04 03:18:48.900568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.900530 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 03:18:48.907086 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:48.907054 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/distinct-2-w4-203c35-kserve-7bb97fcf8-2v6ft"] Nov 04 03:18:49.160782 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:49.160705 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02459976-aea1-488d-83f7-f2525df146ec" path="/var/lib/kubelet/pods/02459976-aea1-488d-83f7-f2525df146ec/volumes" Nov 04 03:18:49.161172 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:49.161108 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" path="/var/lib/kubelet/pods/1b298ff7-1c44-4582-a93c-c3655e1778cb/volumes" Nov 04 03:18:49.161533 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:49.161511 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" path="/var/lib/kubelet/pods/464cc908-f59f-4ce3-9dd1-96a11ebf3188/volumes" Nov 04 03:18:49.162137 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:18:49.162122 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" path="/var/lib/kubelet/pods/7ab74e6d-4c65-42aa-99d1-42e8146a943f/volumes" Nov 04 03:19:01.491482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491443 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:01.491920 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491905 4784 cpu_manager.go:414] "containerMap: removing container" podUID="28ba441a-2429-4d63-b622-9bc9965c524d" containerName="revoke-keys" Nov 04 03:19:01.491966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491922 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ba441a-2429-4d63-b622-9bc9965c524d" containerName="revoke-keys" Nov 04 03:19:01.491966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491948 4784 cpu_manager.go:414] "containerMap: removing container" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerName="payload-pre-processing" Nov 04 03:19:01.491966 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491956 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerName="payload-pre-processing" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491971 4784 cpu_manager.go:414] "containerMap: removing container" podUID="02459976-aea1-488d-83f7-f2525df146ec" containerName="payload-processing" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.491977 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02459976-aea1-488d-83f7-f2525df146ec" containerName="payload-processing" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492000 4784 cpu_manager.go:414] "containerMap: removing container" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492004 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492013 4784 cpu_manager.go:414] "containerMap: removing container" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" Nov 04 03:19:01.492072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492017 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" Nov 04 03:19:01.492337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492083 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b298ff7-1c44-4582-a93c-c3655e1778cb" containerName="main" Nov 04 03:19:01.492337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492091 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="464cc908-f59f-4ce3-9dd1-96a11ebf3188" containerName="main" Nov 04 03:19:01.492337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492097 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ba441a-2429-4d63-b622-9bc9965c524d" containerName="revoke-keys" Nov 04 03:19:01.492337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492104 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab74e6d-4c65-42aa-99d1-42e8146a943f" containerName="payload-pre-processing" Nov 04 03:19:01.492337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.492111 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="02459976-aea1-488d-83f7-f2525df146ec" containerName="payload-processing" Nov 04 03:19:01.495495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.495474 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.529605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529567 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmgq\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.529605 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529619 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.529859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529650 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.529859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529784 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.529859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529832 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.530003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529912 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.530003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.529947 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.530104 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.530003 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.530104 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.530063 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.549314 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.549278 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-203c35-openshift-default-dockercfg-ggkdg\"" Nov 04 03:19:01.604677 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.604646 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:01.630522 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630488 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmgq\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630542 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630566 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630599 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630624 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630657 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.630706 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630686 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630745 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.630802 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631157 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.631125 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631238 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.631178 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.631371 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631460 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.631446 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.631651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.631622 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.633281 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.633258 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.633729 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.633706 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.693294 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.693248 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.693455 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.693365 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmgq\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:01.809243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:01.809112 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:02.240749 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.240719 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:02.241536 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:19:02.241510 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49ab2c7_fe8f_4c72_b5e7_3d550d4638fe.slice/crio-c55c20e2ce7a0c28c9b5e96caf9fe34c4783512c301c929622ab0248a260a54d WatchSource:0}: Error finding container c55c20e2ce7a0c28c9b5e96caf9fe34c4783512c301c929622ab0248a260a54d: Status 404 returned error can't find the container with id c55c20e2ce7a0c28c9b5e96caf9fe34c4783512c301c929622ab0248a260a54d Nov 04 03:19:02.243691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.243655 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:02.243807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.243737 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:02.243807 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.243782 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:02.595902 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.595801 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" event={"ID":"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe","Type":"ContainerStarted","Data":"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3"} Nov 04 03:19:02.595902 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.595899 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" event={"ID":"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe","Type":"ContainerStarted","Data":"c55c20e2ce7a0c28c9b5e96caf9fe34c4783512c301c929622ab0248a260a54d"} Nov 04 03:19:02.726889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.726823 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podStartSLOduration=1.7268052649999999 podStartE2EDuration="1.726805265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:19:02.723418265 +0000 UTC m=+2789.689919260" watchObservedRunningTime="2026-11-04 03:19:02.726805265 +0000 UTC m=+2789.693306257" Nov 04 03:19:02.810160 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:02.810105 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:03.811699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:03.811662 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 04 03:19:03.812139 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:03.811719 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 04 03:19:04.809830 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:04.809794 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 04 03:19:04.810043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:04.809861 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 04 03:19:05.809684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:05.809643 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 04 03:19:05.810150 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:05.809701 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 04 03:19:06.410045 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:19:06.410003 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:19:06.810454 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:06.810403 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 04 03:19:06.811020 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:06.810472 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 04 03:19:07.049424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:07.049386 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92"] Nov 04 03:19:07.100326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:07.100239 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c8ceee-rzq92"] Nov 04 03:19:07.183651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:07.183608 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fecbb7-500a-4adc-b321-a0234f9d21f1" path="/var/lib/kubelet/pods/71fecbb7-500a-4adc-b321-a0234f9d21f1/volumes" Nov 04 03:19:07.809974 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:07.809934 4784 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 04 03:19:07.810177 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:07.809996 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 04 03:19:08.813878 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:08.813844 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:08.814451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:08.814247 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:08.815015 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:08.815000 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:14.691227 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:14.691153 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:14.691778 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:14.691498 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" containerID="cri-o://3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3" gracePeriod=30 Nov 04 03:19:19.946415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:19.946384 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:20.039319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039177 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039281 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039313 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039342 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039401 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039435 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039483 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039571 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039641 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039602 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\") pod \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\" (UID: \"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe\") " Nov 04 03:19:20.039924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039617 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:19:20.039924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039665 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:19:20.039924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039613 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:19:20.039924 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039691 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:19:20.040140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039980 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.040140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.039996 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.040140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.040010 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.040140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.040026 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.040140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.040097 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:19:20.041926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.041900 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:19:20.041926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.041913 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:19:20.041926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.041918 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:19:20.042127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.042017 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq" pod "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" (UID: "c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe"). InnerVolumeSpecName "kube-api-access-hsmgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:19:20.140582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.140543 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.140582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.140573 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.140582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.140581 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.140582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.140592 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmgq\" (UniqueName: \"kubernetes.io/projected/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-kube-api-access-hsmgq\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.140854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.140600 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.177813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.177778 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 03:19:20.178014 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.177984 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78787b6955-88mxk" podUID="e814c148-d53f-4f7c-9728-81d1f504652b" containerName="manager" containerID="cri-o://de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92" gracePeriod=30 Nov 04 03:19:20.427832 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.427803 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 03:19:20.544843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.544807 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\" (UniqueName: \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\") pod \"e814c148-d53f-4f7c-9728-81d1f504652b\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " Nov 04 03:19:20.545043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.544867 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\") pod \"e814c148-d53f-4f7c-9728-81d1f504652b\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " Nov 04 03:19:20.545043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.544947 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\") pod \"e814c148-d53f-4f7c-9728-81d1f504652b\" (UID: \"e814c148-d53f-4f7c-9728-81d1f504652b\") " Nov 04 03:19:20.547220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.547122 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs" pod "e814c148-d53f-4f7c-9728-81d1f504652b" (UID: "e814c148-d53f-4f7c-9728-81d1f504652b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:19:20.547220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.547149 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl" pod "e814c148-d53f-4f7c-9728-81d1f504652b" (UID: "e814c148-d53f-4f7c-9728-81d1f504652b"). InnerVolumeSpecName "kube-api-access-q6bpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:19:20.547220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.547150 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert" pod "e814c148-d53f-4f7c-9728-81d1f504652b" (UID: "e814c148-d53f-4f7c-9728-81d1f504652b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:19:20.646586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.646550 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bpl\" (UniqueName: \"kubernetes.io/projected/e814c148-d53f-4f7c-9728-81d1f504652b-kube-api-access-q6bpl\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.646586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.646585 4784 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-webhook-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.646586 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.646594 4784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e814c148-d53f-4f7c-9728-81d1f504652b-metrics-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:20.711416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.711378 4784 generic.go:356] "Generic (PLEG): container finished" podID="e814c148-d53f-4f7c-9728-81d1f504652b" containerID="de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92" exitCode=0 Nov 04 03:19:20.711416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.711417 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-88mxk" Nov 04 03:19:20.711694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.711473 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-88mxk" event={"ID":"e814c148-d53f-4f7c-9728-81d1f504652b","Type":"ContainerDied","Data":"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92"} Nov 04 03:19:20.711694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.711573 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-88mxk" event={"ID":"e814c148-d53f-4f7c-9728-81d1f504652b","Type":"ContainerDied","Data":"ea5a45fd9057b6515e9b5094362f400634b3367cb29229a7480b12f32dbcf514"} Nov 04 03:19:20.711694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.711604 4784 scope.go:122] "RemoveContainer" containerID="de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92" Nov 04 03:19:20.713162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.713135 4784 generic.go:356] "Generic (PLEG): container finished" podID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerID="3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3" exitCode=0 Nov 04 03:19:20.713325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.713218 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" event={"ID":"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe","Type":"ContainerDied","Data":"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3"} Nov 04 03:19:20.713325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.713275 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" event={"ID":"c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe","Type":"ContainerDied","Data":"c55c20e2ce7a0c28c9b5e96caf9fe34c4783512c301c929622ab0248a260a54d"} Nov 04 03:19:20.713325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.713232 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p" Nov 04 03:19:20.730030 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.730007 4784 scope.go:122] "RemoveContainer" containerID="de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92" Nov 04 03:19:20.730408 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:19:20.730380 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92\": container with ID starting with de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92 not found: ID does not exist" containerID="de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92" Nov 04 03:19:20.730466 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.730439 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92"} err="failed to get container status \"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92\": rpc error: code = NotFound desc = could not find container \"de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92\": container with ID starting with de59e095341298c0fa3d3953fe99c1e36b0ff75eb79c10d4b90e7bee163a8a92 not found: ID does not exist" Nov 04 03:19:20.730511 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.730464 4784 scope.go:122] "RemoveContainer" containerID="3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3" Nov 04 03:19:20.742952 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.742931 4784 scope.go:122] "RemoveContainer" containerID="3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3" Nov 04 03:19:20.743382 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:19:20.743346 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3\": container with ID starting with 3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3 not found: ID does not exist" containerID="3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3" Nov 04 03:19:20.743471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.743395 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3"} err="failed to get container status \"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3\": rpc error: code = NotFound desc = could not find container \"3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3\": container with ID starting with 3fc01d1acfcda2e5bfd363188f3360fb0c5181564b4ae3b138104242489a5fd3 not found: ID does not exist" Nov 04 03:19:20.880236 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.880114 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 03:19:20.885453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.885419 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78787b6955-88mxk"] Nov 04 03:19:20.997087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:20.997049 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:21.000926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.000897 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ssz9p"] Nov 04 03:19:21.160495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.160405 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" path="/var/lib/kubelet/pods/c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe/volumes" Nov 04 03:19:21.160894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.160873 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e814c148-d53f-4f7c-9728-81d1f504652b" path="/var/lib/kubelet/pods/e814c148-d53f-4f7c-9728-81d1f504652b/volumes" Nov 04 03:19:21.759567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.759531 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:21.759969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.759956 4784 cpu_manager.go:414] "containerMap: removing container" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" Nov 04 03:19:21.760029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.759971 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" Nov 04 03:19:21.760029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.759994 4784 cpu_manager.go:414] "containerMap: removing container" podUID="e814c148-d53f-4f7c-9728-81d1f504652b" containerName="manager" Nov 04 03:19:21.760029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.759999 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e814c148-d53f-4f7c-9728-81d1f504652b" containerName="manager" Nov 04 03:19:21.760141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.760075 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e814c148-d53f-4f7c-9728-81d1f504652b" containerName="manager" Nov 04 03:19:21.760141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.760090 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49ab2c7-fe8f-4c72-b5e7-3d550d4638fe" containerName="istio-proxy" Nov 04 03:19:21.764682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.764663 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.822910 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.822877 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:21.859073 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.859045 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl5d\" (UniqueName: \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.859299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.859078 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.859299 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.859247 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.870520 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.870494 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7mq2\"" Nov 04 03:19:21.870694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.870601 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 03:19:21.870694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.870636 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:19:21.870797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.870706 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 03:19:21.960058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.960012 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.960264 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.960108 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl5d\" (UniqueName: \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.960264 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.960137 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.962651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.962625 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:21.962762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:21.962731 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:22.023353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:22.023277 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl5d\" (UniqueName: \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\") pod \"maas-controller-78787b6955-k5gkl\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:22.078317 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:22.078277 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:22.503711 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:22.503683 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:22.504676 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:19:22.504642 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cca4515_adf6_4083_a26d_f449bfcf5275.slice/crio-1bf574b9e7738231d4728d46fc67ce95dd65029a42cf02a58185a89a4623d329 WatchSource:0}: Error finding container 1bf574b9e7738231d4728d46fc67ce95dd65029a42cf02a58185a89a4623d329: Status 404 returned error can't find the container with id 1bf574b9e7738231d4728d46fc67ce95dd65029a42cf02a58185a89a4623d329 Nov 04 03:19:22.506053 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:22.506037 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:19:22.728854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:22.728809 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-k5gkl" event={"ID":"8cca4515-adf6-4083-a26d-f449bfcf5275","Type":"ContainerStarted","Data":"1bf574b9e7738231d4728d46fc67ce95dd65029a42cf02a58185a89a4623d329"} Nov 04 03:19:23.736972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:23.736934 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-k5gkl" event={"ID":"8cca4515-adf6-4083-a26d-f449bfcf5275","Type":"ContainerStarted","Data":"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb"} Nov 04 03:19:23.737490 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:23.737030 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:23.858270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:23.858210 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78787b6955-k5gkl" podStartSLOduration=1.9568936369999999 podStartE2EDuration="2.858172749s" totalImagesPullingTime="901.279112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:19:23.857556335 +0000 UTC m=+2810.824057518" watchObservedRunningTime="2026-11-04 03:19:23.858172749 +0000 UTC m=+2810.824673740" Nov 04 03:19:34.746516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:34.746484 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:36.399772 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:19:36.399731 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:19:36.791817 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:36.791783 4784 scope.go:122] "RemoveContainer" containerID="dc955c4c77f306b444048e24b93fc11802c38893fb727abeb460368d26409ad9" Nov 04 03:19:36.802023 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:36.802001 4784 scope.go:122] "RemoveContainer" containerID="0b50b0474742bfab43528e56f5f76dbd304ac021aba120c03c015e85898646e7" Nov 04 03:19:47.031137 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.031045 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb"] Nov 04 03:19:47.036217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.036173 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107113 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107154 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107181 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107332 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107461 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107396 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7bk\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-kube-api-access-nx7bk\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107467 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107495 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107541 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.107707 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.107638 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.143970 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.143927 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb"] Nov 04 03:19:47.208925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.208862 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.208951 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.208978 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209012 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209048 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209090 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7bk\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-kube-api-access-nx7bk\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209149 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209125 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209356 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209444 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209484 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209456 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209525 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209902 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209875 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.209976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.209959 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.211479 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.211458 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.211625 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.211605 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.271804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.271769 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.271967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.271878 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7bk\" (UniqueName: \"kubernetes.io/projected/83ddc4ac-b046-42be-abf4-d5760f81e9d0-kube-api-access-nx7bk\") pod \"maas-default-gateway-openshift-default-76597d6855-bdgtb\" (UID: \"83ddc4ac-b046-42be-abf4-d5760f81e9d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.350026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.349939 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:47.780969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.780938 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb"] Nov 04 03:19:47.781248 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:19:47.781225 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ddc4ac_b046_42be_abf4_d5760f81e9d0.slice/crio-941b0b22102963e67d6332cede65ad34e5ab9f7e5cd20a22eb0d191cd8b6e875 WatchSource:0}: Error finding container 941b0b22102963e67d6332cede65ad34e5ab9f7e5cd20a22eb0d191cd8b6e875: Status 404 returned error can't find the container with id 941b0b22102963e67d6332cede65ad34e5ab9f7e5cd20a22eb0d191cd8b6e875 Nov 04 03:19:47.783574 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.783530 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:47.783649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.783601 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:47.783649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.783634 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:19:47.894530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.894480 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" event={"ID":"83ddc4ac-b046-42be-abf4-d5760f81e9d0","Type":"ContainerStarted","Data":"9cc89a4f4966d665cba6b0169c87245426d95a77a0c12f3b42d405bb36da9bdd"} Nov 04 03:19:47.894726 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:47.894565 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" event={"ID":"83ddc4ac-b046-42be-abf4-d5760f81e9d0","Type":"ContainerStarted","Data":"941b0b22102963e67d6332cede65ad34e5ab9f7e5cd20a22eb0d191cd8b6e875"} Nov 04 03:19:48.020055 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:48.019984 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podStartSLOduration=2.019962948 podStartE2EDuration="2.019962948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:19:48.017369547 +0000 UTC m=+2834.983870540" watchObservedRunningTime="2026-11-04 03:19:48.019962948 +0000 UTC m=+2834.986463941" Nov 04 03:19:48.350718 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:48.350684 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:48.352487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:48.352454 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:48.352659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:48.352513 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:49.350871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:49.350833 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:49.351335 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:49.350887 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:50.351207 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:50.351155 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:50.351662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:50.351245 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:51.352916 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:51.352871 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:51.353381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:51.352934 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:52.351359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:52.351321 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:52.351690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:52.351390 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:53.350965 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:53.350929 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:53.351445 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:53.350981 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:54.350891 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:54.350857 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:54.351084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:54.350912 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:55.351293 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:55.351256 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 04 03:19:55.351736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:55.351314 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 04 03:19:56.469919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.469881 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:56.470444 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.470109 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78787b6955-k5gkl" podUID="8cca4515-adf6-4083-a26d-f449bfcf5275" containerName="manager" containerID="cri-o://fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb" gracePeriod=30 Nov 04 03:19:56.738852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.738777 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:56.831472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.831431 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\" (UniqueName: \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\") pod \"8cca4515-adf6-4083-a26d-f449bfcf5275\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " Nov 04 03:19:56.831659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.831500 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\") pod \"8cca4515-adf6-4083-a26d-f449bfcf5275\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " Nov 04 03:19:56.831659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.831625 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\") pod \"8cca4515-adf6-4083-a26d-f449bfcf5275\" (UID: \"8cca4515-adf6-4083-a26d-f449bfcf5275\") " Nov 04 03:19:56.833862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.833826 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs" pod "8cca4515-adf6-4083-a26d-f449bfcf5275" (UID: "8cca4515-adf6-4083-a26d-f449bfcf5275"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:19:56.833862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.833843 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert" pod "8cca4515-adf6-4083-a26d-f449bfcf5275" (UID: "8cca4515-adf6-4083-a26d-f449bfcf5275"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:19:56.834030 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.833830 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d" pod "8cca4515-adf6-4083-a26d-f449bfcf5275" (UID: "8cca4515-adf6-4083-a26d-f449bfcf5275"). InnerVolumeSpecName "kube-api-access-zcl5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:19:56.932616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.932582 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl5d\" (UniqueName: \"kubernetes.io/projected/8cca4515-adf6-4083-a26d-f449bfcf5275-kube-api-access-zcl5d\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:56.932616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.932613 4784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-metrics-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:56.932616 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.932623 4784 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cca4515-adf6-4083-a26d-f449bfcf5275-webhook-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:56.957021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.956976 4784 generic.go:356] "Generic (PLEG): container finished" podID="8cca4515-adf6-4083-a26d-f449bfcf5275" containerID="fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb" exitCode=0 Nov 04 03:19:56.957246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.957043 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-k5gkl" Nov 04 03:19:56.957246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.957039 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-k5gkl" event={"ID":"8cca4515-adf6-4083-a26d-f449bfcf5275","Type":"ContainerDied","Data":"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb"} Nov 04 03:19:56.957246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.957179 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-k5gkl" event={"ID":"8cca4515-adf6-4083-a26d-f449bfcf5275","Type":"ContainerDied","Data":"1bf574b9e7738231d4728d46fc67ce95dd65029a42cf02a58185a89a4623d329"} Nov 04 03:19:56.957442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.957291 4784 scope.go:122] "RemoveContainer" containerID="fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb" Nov 04 03:19:56.972509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.972480 4784 scope.go:122] "RemoveContainer" containerID="fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb" Nov 04 03:19:56.972833 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:19:56.972810 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb\": container with ID starting with fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb not found: ID does not exist" containerID="fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb" Nov 04 03:19:56.972919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:56.972841 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb"} err="failed to get container status \"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb\": rpc error: code = NotFound desc = could not find container \"fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb\": container with ID starting with fe9e72a2137391a6a0e225ed59c8eb8d7b65074f9ee6bcd59ed01d48af3fe2bb not found: ID does not exist" Nov 04 03:19:57.128290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:57.128199 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:57.131901 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:57.131871 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78787b6955-k5gkl"] Nov 04 03:19:57.160838 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:57.160446 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cca4515-adf6-4083-a26d-f449bfcf5275" path="/var/lib/kubelet/pods/8cca4515-adf6-4083-a26d-f449bfcf5275/volumes" Nov 04 03:19:57.350365 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:57.350327 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:19:57.350552 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:57.350397 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:19:58.392430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.392383 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:19:58.392860 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.392481 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:19:58.555242 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.555203 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78787b6955-hskxr"] Nov 04 03:19:58.555861 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.555840 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8cca4515-adf6-4083-a26d-f449bfcf5275" containerName="manager" Nov 04 03:19:58.555927 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.555866 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cca4515-adf6-4083-a26d-f449bfcf5275" containerName="manager" Nov 04 03:19:58.556036 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.556024 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cca4515-adf6-4083-a26d-f449bfcf5275" containerName="manager" Nov 04 03:19:58.561045 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.561020 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.615489 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.615452 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 03:19:58.650028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.649939 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-webhook-cert\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.650028 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.649986 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccnw\" (UniqueName: \"kubernetes.io/projected/e82a765d-f8dc-428e-bce3-7d35821d8944-kube-api-access-jccnw\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.650297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.650121 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-metrics-certs\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.666173 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.666137 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 03:19:58.666425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.666395 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:19:58.666583 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.666501 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7mq2\"" Nov 04 03:19:58.671457 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.671421 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-hskxr"] Nov 04 03:19:58.751034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.750993 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-metrics-certs\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.751246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.751072 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-webhook-cert\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.751246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.751099 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccnw\" (UniqueName: \"kubernetes.io/projected/e82a765d-f8dc-428e-bce3-7d35821d8944-kube-api-access-jccnw\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.753699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.753672 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-metrics-certs\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.753842 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.753819 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e82a765d-f8dc-428e-bce3-7d35821d8944-webhook-cert\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.814416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.814378 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccnw\" (UniqueName: \"kubernetes.io/projected/e82a765d-f8dc-428e-bce3-7d35821d8944-kube-api-access-jccnw\") pod \"maas-controller-78787b6955-hskxr\" (UID: \"e82a765d-f8dc-428e-bce3-7d35821d8944\") " pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:58.873587 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.873548 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:58.874092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.874068 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:58.874883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.874863 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" Nov 04 03:19:58.875969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:58.875954 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:19:59.253988 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:59.253958 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78787b6955-hskxr"] Nov 04 03:19:59.254813 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:19:59.254785 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82a765d_f8dc_428e_bce3_7d35821d8944.slice/crio-f2927a3f4c671f17c4c4020df16503f3a6d47b02fc375876eab157d3670d1eb1 WatchSource:0}: Error finding container f2927a3f4c671f17c4c4020df16503f3a6d47b02fc375876eab157d3670d1eb1: Status 404 returned error can't find the container with id f2927a3f4c671f17c4c4020df16503f3a6d47b02fc375876eab157d3670d1eb1 Nov 04 03:19:59.979662 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:19:59.979624 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-hskxr" event={"ID":"e82a765d-f8dc-428e-bce3-7d35821d8944","Type":"ContainerStarted","Data":"f2927a3f4c671f17c4c4020df16503f3a6d47b02fc375876eab157d3670d1eb1"} Nov 04 03:20:00.987495 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:00.987460 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78787b6955-hskxr" event={"ID":"e82a765d-f8dc-428e-bce3-7d35821d8944","Type":"ContainerStarted","Data":"52a9e20fe2cddf2626ff32f0d736cc5df84410de3e6806a44add35b148642b7d"} Nov 04 03:20:00.987973 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:00.987589 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:20:01.110362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:01.110306 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78787b6955-hskxr" podStartSLOduration=2.267067983 podStartE2EDuration="3.110289688s" totalImagesPullingTime="843.221705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:20:01.108833079 +0000 UTC m=+2848.075334068" watchObservedRunningTime="2026-11-04 03:20:01.110289688 +0000 UTC m=+2848.076790727" Nov 04 03:20:02.164267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.164221 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw"] Nov 04 03:20:02.168083 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.168057 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.276297 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.276262 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw"] Nov 04 03:20:02.287661 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.287631 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.287792 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.287710 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bws2\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-kube-api-access-4bws2\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.287792 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.287746 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.287899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.287871 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.288048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.288027 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3303-97e5-4a75-9a2e-05a4b2aac450-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.288124 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.288098 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.288216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.288130 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.288291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.288258 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.288291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.288289 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389315 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389275 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389315 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389317 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389340 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389463 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389517 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389616 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bws2\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-kube-api-access-4bws2\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389659 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389684 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389716 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.389756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389728 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3303-97e5-4a75-9a2e-05a4b2aac450-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.390047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389802 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.390047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389892 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.390047 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.389989 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.390422 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.390388 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3303-97e5-4a75-9a2e-05a4b2aac450-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.392233 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.392203 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.392390 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.392373 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.452785 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.452713 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.452932 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.452779 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bws2\" (UniqueName: \"kubernetes.io/projected/833f3303-97e5-4a75-9a2e-05a4b2aac450-kube-api-access-4bws2\") pod \"maas-default-gateway-openshift-default-76597d6855-v59qw\" (UID: \"833f3303-97e5-4a75-9a2e-05a4b2aac450\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.481720 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.481686 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:02.914486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.914451 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw"] Nov 04 03:20:02.915045 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:20:02.915012 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f3303_97e5_4a75_9a2e_05a4b2aac450.slice/crio-6e5e5e1abbbf17b9240835e2ff5718a0738ad4113646825614bb36e7ec5170f6 WatchSource:0}: Error finding container 6e5e5e1abbbf17b9240835e2ff5718a0738ad4113646825614bb36e7ec5170f6: Status 404 returned error can't find the container with id 6e5e5e1abbbf17b9240835e2ff5718a0738ad4113646825614bb36e7ec5170f6 Nov 04 03:20:02.917340 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.917308 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:20:02.917418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.917374 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:20:02.917418 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:02.917403 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:20:03.000066 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.000026 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" event={"ID":"833f3303-97e5-4a75-9a2e-05a4b2aac450","Type":"ContainerStarted","Data":"fbc9d89b4cddddbd129b4ff9f5be8c735001ae38e68c2125c6c3929c47237480"} Nov 04 03:20:03.000252 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.000097 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" event={"ID":"833f3303-97e5-4a75-9a2e-05a4b2aac450","Type":"ContainerStarted","Data":"6e5e5e1abbbf17b9240835e2ff5718a0738ad4113646825614bb36e7ec5170f6"} Nov 04 03:20:03.128618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.128567 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podStartSLOduration=1.128551399 podStartE2EDuration="1.128551399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:20:03.126282766 +0000 UTC m=+2850.092783758" watchObservedRunningTime="2026-11-04 03:20:03.128551399 +0000 UTC m=+2850.095052396" Nov 04 03:20:03.483214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.482689 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:03.484632 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.484598 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:03.484790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:03.484663 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:04.482604 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:04.482564 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:04.482817 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:04.482622 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:05.482834 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:05.482794 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:05.483307 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:05.482853 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:06.409978 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:20:06.409924 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:20:06.482728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:06.482694 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:06.482968 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:06.482749 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:07.482473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:07.482429 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:07.482741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:07.482501 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:08.482755 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:08.482713 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:08.483353 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:08.482782 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:09.482453 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:09.482415 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:09.482674 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:09.482488 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:10.482895 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:10.482860 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 04 03:20:10.483452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:10.482925 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 04 03:20:11.489339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:11.489305 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:11.489836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:11.489813 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:11.490649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:11.490623 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" Nov 04 03:20:11.995318 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:11.995291 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78787b6955-hskxr" Nov 04 03:20:36.399638 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:20:36.399590 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:20:36.855436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:36.855400 4784 scope.go:122] "RemoveContainer" containerID="d8991027ed5bbecbb712de8e67d6c153d2bfeab6d38f6c8727d27138bdd07d3b" Nov 04 03:20:36.864765 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:36.864743 4784 scope.go:122] "RemoveContainer" containerID="6186537a435cfb3c8356784a8de09882b181c0f3ddc43d58e3bded9882ff5757" Nov 04 03:20:44.922331 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:44.922285 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:20:44.923130 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:44.923093 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:20:53.048962 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:53.048918 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs"] Nov 04 03:20:53.102491 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:53.102452 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d69779-w2mvs"] Nov 04 03:20:53.159168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:20:53.159136 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afed593-cf42-4024-a89f-a8478f85248d" path="/var/lib/kubelet/pods/8afed593-cf42-4024-a89f-a8478f85248d/volumes" Nov 04 03:21:02.639923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.639879 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc"] Nov 04 03:21:02.643692 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.643670 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.735548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.735508 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc"] Nov 04 03:21:02.838307 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838254 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838544 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838420 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838544 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838462 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838544 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838499 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838562 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppgq\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-kube-api-access-zppgq\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838637 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838664 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3813f6c6-ecdf-4990-9be2-fb579378bf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838745 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.838915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.838803 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939761 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939717 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939802 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939831 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939866 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939893 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppgq\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-kube-api-access-zppgq\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939919 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.939979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939941 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3813f6c6-ecdf-4990-9be2-fb579378bf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.940399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.939985 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.940399 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.940039 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.940894 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.940867 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3813f6c6-ecdf-4990-9be2-fb579378bf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.941021 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.940992 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.941081 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.941026 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.941081 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.941061 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.941164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.941084 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.942488 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.942470 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:02.942732 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:02.942716 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:03.003600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.003046 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppgq\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-kube-api-access-zppgq\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:03.003600 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.003545 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3813f6c6-ecdf-4990-9be2-fb579378bf66-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-j8cqc\" (UID: \"3813f6c6-ecdf-4990-9be2-fb579378bf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:03.256378 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.256272 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:03.702278 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.702238 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc"] Nov 04 03:21:03.703346 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:21:03.703320 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3813f6c6_ecdf_4990_9be2_fb579378bf66.slice/crio-a39d2566b21bc7518212d69eab441128a9211a92435eb38a277106ec43c872c7 WatchSource:0}: Error finding container a39d2566b21bc7518212d69eab441128a9211a92435eb38a277106ec43c872c7: Status 404 returned error can't find the container with id a39d2566b21bc7518212d69eab441128a9211a92435eb38a277106ec43c872c7 Nov 04 03:21:03.705571 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.705537 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:21:03.705645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.705605 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:21:03.705645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:03.705632 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:21:04.387401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:04.387360 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" event={"ID":"3813f6c6-ecdf-4990-9be2-fb579378bf66","Type":"ContainerStarted","Data":"942c8221ffd6155511049c435c60ea412c665b3b509b6563fcbaa9ac605662d5"} Nov 04 03:21:04.387596 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:04.387420 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" event={"ID":"3813f6c6-ecdf-4990-9be2-fb579378bf66","Type":"ContainerStarted","Data":"a39d2566b21bc7518212d69eab441128a9211a92435eb38a277106ec43c872c7"} Nov 04 03:21:05.280312 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:05.280268 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:05.280811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:05.280700 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:05.280811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:05.280767 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:06.048877 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:06.048784 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podStartSLOduration=4.048763806 podStartE2EDuration="4.048763806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:04.57546877 +0000 UTC m=+2911.541969763" watchObservedRunningTime="2026-11-04 03:21:06.048763806 +0000 UTC m=+2913.015264799" Nov 04 03:21:06.050589 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:06.050561 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr"] Nov 04 03:21:06.101319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:06.101269 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2953abc-kxcwr"] Nov 04 03:21:06.259738 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:06.259694 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:06.260057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:06.260025 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:06.424016 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:21:06.423980 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:21:07.182216 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:07.181351 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cce5c5-8ab6-4cad-924c-974dd29bc5e2" path="/var/lib/kubelet/pods/f8cce5c5-8ab6-4cad-924c-974dd29bc5e2/volumes" Nov 04 03:21:07.258214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:07.257654 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:07.258214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:07.257725 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:08.258217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:08.257754 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:08.258795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:08.257863 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:09.259221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:09.258820 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:09.259221 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:09.258896 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:10.257260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:10.257162 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:10.257498 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:10.257278 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:11.256862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:11.256824 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 04 03:21:11.257288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:11.256880 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 04 03:21:13.257407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:13.257304 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:21:13.257407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:13.257370 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:21:14.299407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:14.299352 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:21:14.299996 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:14.299436 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:21:14.922998 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:14.922965 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:14.923486 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:14.923457 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:14.924175 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:14.924157 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" Nov 04 03:21:36.399909 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:21:36.399868 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:21:45.997219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:45.989378 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:21:45.997219 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:21:45.989488 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:22:06.400280 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:22:06.400242 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:22:17.469214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:17.458289 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jwwww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:22:17.469214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:17.458358 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jwwww" podUID="f11de3b8-689f-4e98-b418-0a1794cf7907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:22:32.053561 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:32.053512 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk"] Nov 04 03:22:32.102701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:32.102662 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27afeac-n5xrk"] Nov 04 03:22:33.269217 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:33.249692 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54257295-d6fe-4960-8a6b-24b596c82f72" path="/var/lib/kubelet/pods/54257295-d6fe-4960-8a6b-24b596c82f72/volumes" Nov 04 03:22:34.201003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:34.200971 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:22:34.242249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:34.242219 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:22:34.256659 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:34.256631 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:22:34.292762 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:34.292732 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:22:36.399479 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:22:36.399415 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:22:36.905684 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:36.905651 4784 scope.go:122] "RemoveContainer" containerID="41fcc018dc7b62c8288e1b35209d6b0c23b2a46778cb07cba368188d66567947" Nov 04 03:22:36.915084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:36.915065 4784 scope.go:122] "RemoveContainer" containerID="1764e675eeae093581dcbd11a73454f7805e9f6fa920be7776a719d3b68f8433" Nov 04 03:22:49.049826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:49.049786 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm"] Nov 04 03:22:49.100357 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:49.100318 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9d38f41-wmkcm"] Nov 04 03:22:49.159390 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:22:49.159355 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d2e8d1-1d68-45ae-9c7a-426a6ff90b70" path="/var/lib/kubelet/pods/69d2e8d1-1d68-45ae-9c7a-426a6ff90b70/volumes" Nov 04 03:23:06.563215 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:23:06.561417 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:23:30.977443 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:30.977363 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:23:30.978682 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:30.978649 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:23:32.050422 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:32.050386 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42"] Nov 04 03:23:32.101426 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:32.101390 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e54fb7e-xsm42"] Nov 04 03:23:33.159722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:33.159690 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9a4ea3-d64d-4d49-a949-34e1b9d61d81" path="/var/lib/kubelet/pods/dc9a4ea3-d64d-4d49-a949-34e1b9d61d81/volumes" Nov 04 03:23:36.401450 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:23:36.401402 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:23:36.941538 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:36.941504 4784 scope.go:122] "RemoveContainer" containerID="4e8ef6f21999cf1ee45049f01bc508863d3b504afb4981d4b696fd6c837f2221" Nov 04 03:23:45.060359 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:45.060323 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8"] Nov 04 03:23:45.111283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:45.111245 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0eb77b-2h4p8"] Nov 04 03:23:45.160368 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:23:45.160325 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ba441a-2429-4d63-b622-9bc9965c524d" path="/var/lib/kubelet/pods/28ba441a-2429-4d63-b622-9bc9965c524d/volumes" Nov 04 03:24:06.400753 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:24:06.400710 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:24:36.400943 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:24:36.400901 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:24:36.966483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:24:36.966449 4784 scope.go:122] "RemoveContainer" containerID="1bbac6e3473ac9eba7aa33b4433200762a897cff71b236abd1afb08027e0e519" Nov 04 03:24:36.976012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:24:36.975986 4784 scope.go:122] "RemoveContainer" containerID="c68058f04b3583538881a3da5473dfbe856e180e0859f78d4c95db74333508e3" Nov 04 03:25:06.399755 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:25:06.399716 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:25:36.399914 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:25:36.399876 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:25:37.003704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:25:37.003670 4784 scope.go:122] "RemoveContainer" containerID="dff75617fc264226eabe6f8a8811f708fd2ec9ef2cef68c30b96a6ed3ac48bce" Nov 04 03:26:06.400486 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:26:06.400443 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:26:28.385590 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.385554 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:26:28.389393 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.389374 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440490 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440451 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440490 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440492 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440525 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440675 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440877 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440774 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdcl\" (UniqueName: \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.440877 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.440823 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.460972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.460938 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:26:28.496363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.496330 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-07ce81\"/\"simulator-single-main-07ce81-kserve-self-signed-certs\"" Nov 04 03:26:28.496560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.496366 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-07ce81\"/\"openshift-service-ca.crt\"" Nov 04 03:26:28.496560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.496373 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-07ce81\"/\"kube-root-ca.crt\"" Nov 04 03:26:28.496560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.496469 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-07ce81\"/\"default-dockercfg-hfvls\"" Nov 04 03:26:28.542358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542318 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542388 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdcl\" (UniqueName: \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542418 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542495 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542516 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542566 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542562 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542849 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542831 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542878 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.542969 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.542941 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.544914 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.544890 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.545243 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.545227 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.605290 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.605250 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdcl\" (UniqueName: \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\") pod \"simulator-single-main-07ce81-kserve-bf585c59b-hwgbc\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:28.701253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:28.701213 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:29.081220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:29.081158 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:26:29.082006 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:26:29.081976 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod791c591d_6573_43be_a0bd_fd8f0b7b90b8.slice/crio-f71ecda9bb0b57187fcd0c779efb2c1d235f09067598e6ee94040dbd654a34e1 WatchSource:0}: Error finding container f71ecda9bb0b57187fcd0c779efb2c1d235f09067598e6ee94040dbd654a34e1: Status 404 returned error can't find the container with id f71ecda9bb0b57187fcd0c779efb2c1d235f09067598e6ee94040dbd654a34e1 Nov 04 03:26:29.893064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:29.893012 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" event={"ID":"791c591d-6573-43be-a0bd-fd8f0b7b90b8","Type":"ContainerStarted","Data":"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb"} Nov 04 03:26:29.893509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:29.893072 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" event={"ID":"791c591d-6573-43be-a0bd-fd8f0b7b90b8","Type":"ContainerStarted","Data":"f71ecda9bb0b57187fcd0c779efb2c1d235f09067598e6ee94040dbd654a34e1"} Nov 04 03:26:30.015769 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:30.015710 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" podStartSLOduration=2.01569433 podStartE2EDuration="2.01569433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:30.014457349 +0000 UTC m=+3236.980958342" watchObservedRunningTime="2026-11-04 03:26:30.01569433 +0000 UTC m=+3236.982195315" Nov 04 03:26:30.701649 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:30.701604 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:36.401500 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:26:36.401458 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:26:38.702363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:38.702318 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:38.715503 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:38.715470 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:26:38.957037 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:26:38.956943 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:27:06.401227 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:27:06.401160 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:27:13.044473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.044383 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs"] Nov 04 03:27:13.048159 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.048141 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.102640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.102580 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:27:13.102819 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.102658 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:27:13.121337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.121304 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.121505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.121425 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxdj\" (UniqueName: \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.121505 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.121502 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.153450 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.153420 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:27:13.153638 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.153452 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:27:13.159040 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.159015 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs"] Nov 04 03:27:13.222700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.222661 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.222904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.222748 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.222904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.222838 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxdj\" (UniqueName: \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.223548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.223467 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.225257 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.225237 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.285058 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.285027 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxdj\" (UniqueName: \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\") pod \"maas-api-revoke-sub-fbc30d64-cj2cs\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.361298 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.361209 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:13.738132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:13.738098 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs"] Nov 04 03:27:13.739555 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:27:13.739526 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1f6030_a24e_4b9b_905d_cd53b1a6ade2.slice/crio-c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397 WatchSource:0}: Error finding container c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397: Status 404 returned error can't find the container with id c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397 Nov 04 03:27:14.149168 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:14.149126 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" event={"ID":"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2","Type":"ContainerStarted","Data":"402586f3b4ffcf729616446ccd2e60485e665d2d54d55fe0351fceeab737d4af"} Nov 04 03:27:14.149609 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:14.149202 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" event={"ID":"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2","Type":"ContainerStarted","Data":"c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397"} Nov 04 03:27:14.267138 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:14.267082 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" podStartSLOduration=1.2670692350000001 podStartE2EDuration="1.267069235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:27:14.266731318 +0000 UTC m=+3281.233232311" watchObservedRunningTime="2026-11-04 03:27:14.267069235 +0000 UTC m=+3281.233570226" Nov 04 03:27:15.157151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:15.157120 4784 generic.go:356] "Generic (PLEG): container finished" podID="9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" containerID="402586f3b4ffcf729616446ccd2e60485e665d2d54d55fe0351fceeab737d4af" exitCode=0 Nov 04 03:27:15.159180 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:15.159156 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" event={"ID":"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2","Type":"ContainerDied","Data":"402586f3b4ffcf729616446ccd2e60485e665d2d54d55fe0351fceeab737d4af"} Nov 04 03:27:16.312591 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.312561 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:16.461939 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.461898 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\" (UniqueName: \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\") pod \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " Nov 04 03:27:16.462200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.461970 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\") pod \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " Nov 04 03:27:16.462200 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.462050 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\") pod \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\" (UID: \"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2\") " Nov 04 03:27:16.462380 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.462355 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca" pod "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" (UID: "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:27:16.462564 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.462545 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:16.464327 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.464303 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca" pod "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" (UID: "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:27:16.464433 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.464386 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj" pod "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" (UID: "9f1f6030-a24e-4b9b-905d-cd53b1a6ade2"). InnerVolumeSpecName "kube-api-access-cbxdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:27:16.563408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.563361 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:16.563408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:16.563406 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxdj\" (UniqueName: \"kubernetes.io/projected/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2-kube-api-access-cbxdj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:17.174446 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:17.174401 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" event={"ID":"9f1f6030-a24e-4b9b-905d-cd53b1a6ade2","Type":"ContainerDied","Data":"c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397"} Nov 04 03:27:17.174647 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:17.174509 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9986fe6ed2279165ec3b316d6b064f4151f7782fce694bd2a325c9f363aa397" Nov 04 03:27:17.174647 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:17.174594 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs" Nov 04 03:27:29.683989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:29.683943 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:27:34.332104 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:34.332069 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:27:34.337601 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:34.337573 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:27:34.435096 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:34.435064 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:27:34.440291 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:34.440268 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:27:36.400489 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:27:36.400450 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:27:44.705521 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:44.705477 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerName="main" containerID="cri-o://ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb" gracePeriod=45 Nov 04 03:27:44.974872 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:44.974842 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:27:45.069922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.069880 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.069922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.069924 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.070164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.069985 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.070164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070010 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.070164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070037 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\" (UniqueName: \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.070164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070082 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\") pod \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\" (UID: \"791c591d-6573-43be-a0bd-fd8f0b7b90b8\") " Nov 04 03:27:45.070363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070180 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:27:45.070407 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070389 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:27:45.070483 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070457 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:27:45.070524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070504 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.070524 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.070515 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.072416 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.072385 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:27:45.072570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.072554 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:27:45.072665 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.072645 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl" pod "791c591d-6573-43be-a0bd-fd8f0b7b90b8" (UID: "791c591d-6573-43be-a0bd-fd8f0b7b90b8"). InnerVolumeSpecName "kube-api-access-fzdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:27:45.171603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.171571 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.171603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.171602 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.171804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.171638 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791c591d-6573-43be-a0bd-fd8f0b7b90b8-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.171804 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.171650 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzdcl\" (UniqueName: \"kubernetes.io/projected/791c591d-6573-43be-a0bd-fd8f0b7b90b8-kube-api-access-fzdcl\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:45.334500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.334395 4784 generic.go:356] "Generic (PLEG): container finished" podID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerID="ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb" exitCode=0 Nov 04 03:27:45.334500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.334448 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" Nov 04 03:27:45.334500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.334479 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" event={"ID":"791c591d-6573-43be-a0bd-fd8f0b7b90b8","Type":"ContainerDied","Data":"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb"} Nov 04 03:27:45.334777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.334535 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" event={"ID":"791c591d-6573-43be-a0bd-fd8f0b7b90b8","Type":"ContainerDied","Data":"f71ecda9bb0b57187fcd0c779efb2c1d235f09067598e6ee94040dbd654a34e1"} Nov 04 03:27:45.334777 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.334556 4784 scope.go:122] "RemoveContainer" containerID="ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb" Nov 04 03:27:45.345935 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.345905 4784 scope.go:122] "RemoveContainer" containerID="ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb" Nov 04 03:27:45.346325 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:27:45.346292 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb\": container with ID starting with ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb not found: ID does not exist" containerID="ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb" Nov 04 03:27:45.346386 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.346336 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb"} err="failed to get container status \"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb\": rpc error: code = NotFound desc = could not find container \"ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb\": container with ID starting with ada543dab0407d67803e1950a3b29b1623a56d4fa01ed27410a8ae52176ccccb not found: ID does not exist" Nov 04 03:27:45.504708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.504667 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:27:45.508509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.508477 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc"] Nov 04 03:27:45.947402 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:45.947347 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-07ce81/simulator-single-main-07ce81-kserve-bf585c59b-hwgbc" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.240:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:27:47.159475 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:27:47.159443 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" path="/var/lib/kubelet/pods/791c591d-6573-43be-a0bd-fd8f0b7b90b8/volumes" Nov 04 03:28:06.400595 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:28:06.400554 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:28:36.401324 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:28:36.401284 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:29:06.400729 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:29:06.400686 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:29:36.399787 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:29:36.399740 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:29:57.545428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:29:57.545335 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:29:57.546326 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:29:57.546267 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:30:01.863887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.863841 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks"] Nov 04 03:30:01.864384 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864368 4784 cpu_manager.go:414] "containerMap: removing container" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerName="main" Nov 04 03:30:01.864430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864385 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerName="main" Nov 04 03:30:01.864430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864404 4784 cpu_manager.go:414] "containerMap: removing container" podUID="9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" containerName="revoke-keys" Nov 04 03:30:01.864430 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864409 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" containerName="revoke-keys" Nov 04 03:30:01.864530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864492 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="791c591d-6573-43be-a0bd-fd8f0b7b90b8" containerName="main" Nov 04 03:30:01.864530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.864504 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" containerName="revoke-keys" Nov 04 03:30:01.867909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.867885 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:01.921823 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.921785 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:30:01.926355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.926326 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:01.926454 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.926371 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:01.926502 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.926478 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdrz\" (UniqueName: \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:01.972646 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.972609 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:30:01.972850 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.972826 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:30:01.972919 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.972827 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:30:01.975694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:01.975668 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks"] Nov 04 03:30:02.027582 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.027539 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.027795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.027608 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdrz\" (UniqueName: \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.027795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.027731 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.028250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.028220 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.030222 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.030183 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.090719 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.090683 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdrz\" (UniqueName: \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\") pod \"maas-api-revoke-sub-d576e68b-2zmks\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.181568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.181532 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:02.560892 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:02.560859 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks"] Nov 04 03:30:02.562251 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:30:02.562184 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e824942_0c48_4404_a2ac_e93762472004.slice/crio-b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78 WatchSource:0}: Error finding container b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78: Status 404 returned error can't find the container with id b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78 Nov 04 03:30:03.223654 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:03.223611 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" event={"ID":"0e824942-0c48-4404-a2ac-e93762472004","Type":"ContainerStarted","Data":"8f79c457e3e67f0e4fdcecafac0f724ad24a862305dbea6ffef78bf4ef5f98d8"} Nov 04 03:30:03.224246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:03.223671 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" event={"ID":"0e824942-0c48-4404-a2ac-e93762472004","Type":"ContainerStarted","Data":"b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78"} Nov 04 03:30:03.342267 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:03.342182 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" podStartSLOduration=2.342168821 podStartE2EDuration="2.342168821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:03.341853617 +0000 UTC m=+3450.308354610" watchObservedRunningTime="2026-11-04 03:30:03.342168821 +0000 UTC m=+3450.308669813" Nov 04 03:30:04.231141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:04.231102 4784 generic.go:356] "Generic (PLEG): container finished" podID="0e824942-0c48-4404-a2ac-e93762472004" containerID="8f79c457e3e67f0e4fdcecafac0f724ad24a862305dbea6ffef78bf4ef5f98d8" exitCode=0 Nov 04 03:30:04.231611 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:04.231180 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" event={"ID":"0e824942-0c48-4404-a2ac-e93762472004","Type":"ContainerDied","Data":"8f79c457e3e67f0e4fdcecafac0f724ad24a862305dbea6ffef78bf4ef5f98d8"} Nov 04 03:30:05.383363 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.383337 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:05.468366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.468326 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\") pod \"0e824942-0c48-4404-a2ac-e93762472004\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " Nov 04 03:30:05.468560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.468446 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\") pod \"0e824942-0c48-4404-a2ac-e93762472004\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " Nov 04 03:30:05.468560 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.468474 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\" (UniqueName: \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\") pod \"0e824942-0c48-4404-a2ac-e93762472004\" (UID: \"0e824942-0c48-4404-a2ac-e93762472004\") " Nov 04 03:30:05.468763 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.468737 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca" pod "0e824942-0c48-4404-a2ac-e93762472004" (UID: "0e824942-0c48-4404-a2ac-e93762472004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:05.470785 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.470753 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca" pod "0e824942-0c48-4404-a2ac-e93762472004" (UID: "0e824942-0c48-4404-a2ac-e93762472004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:30:05.470785 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.470776 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz" pod "0e824942-0c48-4404-a2ac-e93762472004" (UID: "0e824942-0c48-4404-a2ac-e93762472004"). InnerVolumeSpecName "kube-api-access-tfdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:05.570110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.570016 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e824942-0c48-4404-a2ac-e93762472004-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:05.570110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.570055 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e824942-0c48-4404-a2ac-e93762472004-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:05.570110 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:05.570066 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdrz\" (UniqueName: \"kubernetes.io/projected/0e824942-0c48-4404-a2ac-e93762472004-kube-api-access-tfdrz\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:06.243900 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:06.243871 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" Nov 04 03:30:06.244087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:06.243908 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks" event={"ID":"0e824942-0c48-4404-a2ac-e93762472004","Type":"ContainerDied","Data":"b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78"} Nov 04 03:30:06.244087 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:30:06.243960 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b41b6d8f7678c712ef952b0a66b60e33415b371b8997ab8ee9d430637f265e78" Nov 04 03:30:06.401485 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:30:06.401447 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:30:36.400877 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:30:36.400839 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:31:06.403896 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:31:06.403848 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:31:09.742452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.742416 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:31:09.742950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.742887 4784 cpu_manager.go:414] "containerMap: removing container" podUID="0e824942-0c48-4404-a2ac-e93762472004" containerName="revoke-keys" Nov 04 03:31:09.742950 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.742898 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e824942-0c48-4404-a2ac-e93762472004" containerName="revoke-keys" Nov 04 03:31:09.743024 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.742970 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e824942-0c48-4404-a2ac-e93762472004" containerName="revoke-keys" Nov 04 03:31:09.746289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.746272 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.851570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851527 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8clb\" (UniqueName: \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.851790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851583 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.851790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851696 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.851790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851708 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2f48c1\"/\"openshift-service-ca.crt\"" Nov 04 03:31:09.851790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851750 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.851790 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851766 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2f48c1\"/\"kube-root-ca.crt\"" Nov 04 03:31:09.852111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851933 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2f48c1\"/\"simulator-single-main-2f48c1-kserve-self-signed-certs\"" Nov 04 03:31:09.852111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.851967 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.852111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.852005 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.852111 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.852058 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2f48c1\"/\"default-dockercfg-lxntr\"" Nov 04 03:31:09.854772 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.854751 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:31:09.953339 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.953238 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.953979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.953577 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.953979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.953776 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.953979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.953838 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.953979 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.953916 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8clb\" (UniqueName: \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.955354 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.954370 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.955354 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.954773 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.955354 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.955316 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.956323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.956097 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.959687 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.959657 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:09.960032 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:09.960008 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:10.017911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.017817 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8clb\" (UniqueName: \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\") pod \"simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:10.059059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.059028 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:10.438636 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.438610 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:31:10.440151 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:31:10.440122 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4a325f2_f31a_4f76_813f_b2356131ac99.slice/crio-bf97e3fcc27deda3e288eea5689944780dbac8c186ccbf8e8d996912ab50a754 WatchSource:0}: Error finding container bf97e3fcc27deda3e288eea5689944780dbac8c186ccbf8e8d996912ab50a754: Status 404 returned error can't find the container with id bf97e3fcc27deda3e288eea5689944780dbac8c186ccbf8e8d996912ab50a754 Nov 04 03:31:10.785822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.785723 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" event={"ID":"b4a325f2-f31a-4f76-813f-b2356131ac99","Type":"ContainerStarted","Data":"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c"} Nov 04 03:31:10.785822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.785768 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" event={"ID":"b4a325f2-f31a-4f76-813f-b2356131ac99","Type":"ContainerStarted","Data":"bf97e3fcc27deda3e288eea5689944780dbac8c186ccbf8e8d996912ab50a754"} Nov 04 03:31:10.906750 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:10.906696 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" podStartSLOduration=1.906682003 podStartE2EDuration="1.906682003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:31:10.904002078 +0000 UTC m=+3517.870503327" watchObservedRunningTime="2026-11-04 03:31:10.906682003 +0000 UTC m=+3517.873182998" Nov 04 03:31:11.060001 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:11.059910 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:20.060164 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:20.060127 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:20.073048 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:20.073017 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:20.852741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:20.852705 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:31:36.400593 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:31:36.400551 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:31:54.592995 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.592941 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97"] Nov 04 03:31:54.596756 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.596734 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.650862 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.650821 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:31:54.651065 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.650904 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:31:54.701875 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.701837 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:31:54.702060 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.701900 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:31:54.705077 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.705055 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97"] Nov 04 03:31:54.721544 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.721509 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.721708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.721649 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhtv\" (UniqueName: \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.721797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.721713 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.823161 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.823125 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.823415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.823248 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.823415 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.823314 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhtv\" (UniqueName: \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.823959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.823931 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.825708 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.825688 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.885881 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.885800 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhtv\" (UniqueName: \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\") pod \"maas-api-revoke-sub-ecfbe564-mhv97\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:54.910197 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:54.910168 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:55.298198 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:55.296668 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97"] Nov 04 03:31:55.304656 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:31:55.304623 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9e63d9_c187_4668_8fe4_dbb526ccbe92.slice/crio-745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81 WatchSource:0}: Error finding container 745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81: Status 404 returned error can't find the container with id 745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81 Nov 04 03:31:56.051513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:56.051475 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" event={"ID":"4d9e63d9-c187-4668-8fe4-dbb526ccbe92","Type":"ContainerStarted","Data":"02b0bd68224a49f5109defccc1b1b065b09494bf4c78271562e95e578b5c7539"} Nov 04 03:31:56.051911 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:56.051530 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" event={"ID":"4d9e63d9-c187-4668-8fe4-dbb526ccbe92","Type":"ContainerStarted","Data":"745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81"} Nov 04 03:31:56.169767 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:56.169720 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" podStartSLOduration=2.169704498 podStartE2EDuration="2.169704498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:31:56.169301064 +0000 UTC m=+3563.135802061" watchObservedRunningTime="2026-11-04 03:31:56.169704498 +0000 UTC m=+3563.136205493" Nov 04 03:31:57.060851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:57.060798 4784 generic.go:356] "Generic (PLEG): container finished" podID="4d9e63d9-c187-4668-8fe4-dbb526ccbe92" containerID="02b0bd68224a49f5109defccc1b1b065b09494bf4c78271562e95e578b5c7539" exitCode=0 Nov 04 03:31:57.060851 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:57.060845 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" event={"ID":"4d9e63d9-c187-4668-8fe4-dbb526ccbe92","Type":"ContainerDied","Data":"02b0bd68224a49f5109defccc1b1b065b09494bf4c78271562e95e578b5c7539"} Nov 04 03:31:58.218003 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.217979 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:58.360577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.360479 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\") pod \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " Nov 04 03:31:58.360768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.360606 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\" (UniqueName: \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\") pod \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " Nov 04 03:31:58.360768 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.360675 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\") pod \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\" (UID: \"4d9e63d9-c187-4668-8fe4-dbb526ccbe92\") " Nov 04 03:31:58.361078 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.361039 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca" pod "4d9e63d9-c187-4668-8fe4-dbb526ccbe92" (UID: "4d9e63d9-c187-4668-8fe4-dbb526ccbe92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:31:58.363355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.363325 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca" pod "4d9e63d9-c187-4668-8fe4-dbb526ccbe92" (UID: "4d9e63d9-c187-4668-8fe4-dbb526ccbe92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:31:58.363487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.363349 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv" pod "4d9e63d9-c187-4668-8fe4-dbb526ccbe92" (UID: "4d9e63d9-c187-4668-8fe4-dbb526ccbe92"). InnerVolumeSpecName "kube-api-access-rnhtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:31:58.462260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.462180 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnhtv\" (UniqueName: \"kubernetes.io/projected/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-kube-api-access-rnhtv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:31:58.462260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.462255 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:31:58.462260 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:58.462267 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9e63d9-c187-4668-8fe4-dbb526ccbe92-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:31:59.073179 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:59.073148 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" Nov 04 03:31:59.073397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:59.073199 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97" event={"ID":"4d9e63d9-c187-4668-8fe4-dbb526ccbe92","Type":"ContainerDied","Data":"745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81"} Nov 04 03:31:59.073397 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:31:59.073249 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745c8787c53ebcea89c8f91a69478ac28b236272e3e0ca5b08657fdb51ae8f81" Nov 04 03:32:06.399645 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:32:06.399605 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:32:10.690948 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:10.690910 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:32:17.093487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:17.093441 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs"] Nov 04 03:32:17.101452 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:17.101423 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc30d64-cj2cs"] Nov 04 03:32:17.159915 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:17.159879 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1f6030-a24e-4b9b-905d-cd53b1a6ade2" path="/var/lib/kubelet/pods/9f1f6030-a24e-4b9b-905d-cd53b1a6ade2/volumes" Nov 04 03:32:25.712092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:25.712024 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerName="main" containerID="cri-o://c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c" gracePeriod=45 Nov 04 03:32:25.842387 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:25.842345 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerName="main" probeResult="failure" output="Get \"https://10.133.0.243:8000/ready\": dial tcp 10.133.0.243:8000: connect: connection refused" Nov 04 03:32:25.976256 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:25.976150 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:32:26.041456 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041414 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041466 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041549 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041683 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041604 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\" (UniqueName: \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041683 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041740 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\") pod \"b4a325f2-f31a-4f76-813f-b2356131ac99\" (UID: \"b4a325f2-f31a-4f76-813f-b2356131ac99\") " Nov 04 03:32:26.041871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041797 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.041871 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.041834 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.042107 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.042086 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.042175 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.042110 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.042175 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.042107 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.043928 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.043907 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.044018 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.043950 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:32:26.044134 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.044116 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb" pod "b4a325f2-f31a-4f76-813f-b2356131ac99" (UID: "b4a325f2-f31a-4f76-813f-b2356131ac99"). InnerVolumeSpecName "kube-api-access-q8clb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:26.142984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.142948 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.142984 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.142989 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8clb\" (UniqueName: \"kubernetes.io/projected/b4a325f2-f31a-4f76-813f-b2356131ac99-kube-api-access-q8clb\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.143218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.143001 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a325f2-f31a-4f76-813f-b2356131ac99-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.143218 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.143009 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a325f2-f31a-4f76-813f-b2356131ac99-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.230478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.230380 4784 generic.go:356] "Generic (PLEG): container finished" podID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerID="c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c" exitCode=0 Nov 04 03:32:26.230478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.230423 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" Nov 04 03:32:26.230478 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.230463 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" event={"ID":"b4a325f2-f31a-4f76-813f-b2356131ac99","Type":"ContainerDied","Data":"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c"} Nov 04 03:32:26.230779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.230514 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8" event={"ID":"b4a325f2-f31a-4f76-813f-b2356131ac99","Type":"ContainerDied","Data":"bf97e3fcc27deda3e288eea5689944780dbac8c186ccbf8e8d996912ab50a754"} Nov 04 03:32:26.230779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.230535 4784 scope.go:122] "RemoveContainer" containerID="c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c" Nov 04 03:32:26.241088 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.241067 4784 scope.go:122] "RemoveContainer" containerID="c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c" Nov 04 03:32:26.241391 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:32:26.241369 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c\": container with ID starting with c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c not found: ID does not exist" containerID="c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c" Nov 04 03:32:26.241474 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.241399 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c"} err="failed to get container status \"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c\": rpc error: code = NotFound desc = could not find container \"c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c\": container with ID starting with c6dcc31443dfb953a9bcc1ee1aca0d04693494b90a6fae2392c89d78abd66c2c not found: ID does not exist" Nov 04 03:32:26.399602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.399563 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:32:26.404006 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:26.403977 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2f48c1/simulator-single-main-2f48c1-kserve-7c7867c454-7h4s8"] Nov 04 03:32:27.160622 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:27.160586 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" path="/var/lib/kubelet/pods/b4a325f2-f31a-4f76-813f-b2356131ac99/volumes" Nov 04 03:32:34.376065 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:34.376039 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:32:34.381134 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:34.381114 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:32:34.473870 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:34.473834 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:32:34.478968 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:32:34.478947 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:32:36.399526 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:32:36.399484 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:33:00.740598 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.740543 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq"] Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741019 4784 cpu_manager.go:414] "containerMap: removing container" podUID="4d9e63d9-c187-4668-8fe4-dbb526ccbe92" containerName="revoke-keys" Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741031 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9e63d9-c187-4668-8fe4-dbb526ccbe92" containerName="revoke-keys" Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741048 4784 cpu_manager.go:414] "containerMap: removing container" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerName="main" Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741053 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerName="main" Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741120 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a325f2-f31a-4f76-813f-b2356131ac99" containerName="main" Nov 04 03:33:00.741155 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.741136 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9e63d9-c187-4668-8fe4-dbb526ccbe92" containerName="revoke-keys" Nov 04 03:33:00.745776 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.745754 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.801993 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.801962 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq"] Nov 04 03:33:00.803013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.802986 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:33:00.803013 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.803005 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:33:00.808739 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.808708 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.808906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.808800 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cb5z\" (UniqueName: \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.808906 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.808852 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.849521 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.849484 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:33:00.849521 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.849507 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:33:00.910410 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.910370 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cb5z\" (UniqueName: \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.910621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.910436 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.910621 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.910475 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.911114 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.911090 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.912961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.912935 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:00.972822 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:00.972788 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cb5z\" (UniqueName: \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:01.059634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:01.059543 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:01.437482 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:01.437456 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq"] Nov 04 03:33:01.438381 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:33:01.438350 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a0d7ee_4b9c_4a32_a046_e8bc113b6c3e.slice/crio-33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5 WatchSource:0}: Error finding container 33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5: Status 404 returned error can't find the container with id 33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5 Nov 04 03:33:02.437036 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:02.436998 4784 generic.go:356] "Generic (PLEG): container finished" podID="14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" containerID="ce2e3d1c93a591d328daac5ce205329669dbfeb56283f6fdd8f88c600065a5e8" exitCode=0 Nov 04 03:33:02.437500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:02.437075 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" event={"ID":"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e","Type":"ContainerDied","Data":"ce2e3d1c93a591d328daac5ce205329669dbfeb56283f6fdd8f88c600065a5e8"} Nov 04 03:33:02.437500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:02.437136 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" event={"ID":"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e","Type":"ContainerStarted","Data":"33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5"} Nov 04 03:33:03.584351 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.584322 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:03.642283 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.642246 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\" (UniqueName: \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\") pod \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " Nov 04 03:33:03.642468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.642345 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\") pod \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " Nov 04 03:33:03.642468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.642405 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\") pod \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\" (UID: \"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e\") " Nov 04 03:33:03.642770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.642747 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca" pod "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" (UID: "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:33:03.644631 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.644607 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca" pod "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" (UID: "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:33:03.644710 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.644635 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z" pod "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" (UID: "14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e"). InnerVolumeSpecName "kube-api-access-4cb5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:03.744043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.743947 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cb5z\" (UniqueName: \"kubernetes.io/projected/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-kube-api-access-4cb5z\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:03.744043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.743993 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:03.744043 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:03.744009 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:04.460980 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:04.460934 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" Nov 04 03:33:04.461180 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:04.460987 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq" event={"ID":"14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e","Type":"ContainerDied","Data":"33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5"} Nov 04 03:33:04.461180 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:04.461092 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33220cd46b4dab3fc7deb8cfb98405239c57ea0ff491980516a1ab9c24669ca5" Nov 04 03:33:06.400596 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:33:06.400542 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:33:20.619473 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:20.619405 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq"] Nov 04 03:33:20.670457 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:20.670396 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2f48c1-chpqq"] Nov 04 03:33:21.163270 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:21.163234 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" path="/var/lib/kubelet/pods/14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e/volumes" Nov 04 03:33:36.400003 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:33:36.399963 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:33:37.101011 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:37.100976 4784 scope.go:122] "RemoveContainer" containerID="402586f3b4ffcf729616446ccd2e60485e665d2d54d55fe0351fceeab737d4af" Nov 04 03:33:54.904253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.904209 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:33:54.904888 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.904865 4784 cpu_manager.go:414] "containerMap: removing container" podUID="14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" containerName="revoke-keys" Nov 04 03:33:54.904989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.904891 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" containerName="revoke-keys" Nov 04 03:33:54.905050 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.905031 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a0d7ee-4b9c-4a32-a046-e8bc113b6c3e" containerName="revoke-keys" Nov 04 03:33:54.911140 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.911109 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.938694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938652 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.938694 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938695 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.938930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938717 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmp6q\" (UniqueName: \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.938930 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938892 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.939026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938943 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:54.939026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:54.938994 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.028349 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.028313 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b05dd916-kserve-self-signed-certs\"" Nov 04 03:33:55.031119 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.031093 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:33:55.040008 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.039975 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040209 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040014 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040209 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040080 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040209 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040105 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040209 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040135 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmp6q\" (UniqueName: \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040424 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040277 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040468 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040451 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040502 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.040589 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.040572 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.042797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.042773 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.043027 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.043001 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.102896 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.102866 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmp6q\" (UniqueName: \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\") pod \"e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.225062 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.225017 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:33:55.602861 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.602827 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:33:55.603950 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:33:55.603922 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe173d02_3479_44b4_97b3_78142c757a80.slice/crio-d70701ffcdfda0eb6395d1182215d069711bdc8db7773d97c47b291484eee154 WatchSource:0}: Error finding container d70701ffcdfda0eb6395d1182215d069711bdc8db7773d97c47b291484eee154: Status 404 returned error can't find the container with id d70701ffcdfda0eb6395d1182215d069711bdc8db7773d97c47b291484eee154 Nov 04 03:33:55.848934 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.848891 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" event={"ID":"be173d02-3479-44b4-97b3-78142c757a80","Type":"ContainerStarted","Data":"08ac002fc1bdff99431b164fa96ba1d75fb37ff3eeb5f4c993de4fefc9e4b6c8"} Nov 04 03:33:55.848934 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.848935 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" event={"ID":"be173d02-3479-44b4-97b3-78142c757a80","Type":"ContainerStarted","Data":"d70701ffcdfda0eb6395d1182215d069711bdc8db7773d97c47b291484eee154"} Nov 04 03:33:55.970249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:55.970179 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" podStartSLOduration=1.970165583 podStartE2EDuration="1.970165583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:55.967757835 +0000 UTC m=+3682.934258830" watchObservedRunningTime="2026-11-04 03:33:55.970165583 +0000 UTC m=+3682.936666575" Nov 04 03:33:56.225700 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:33:56.225606 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:34:05.225432 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:05.225389 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:34:05.237903 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:05.237874 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:34:05.968961 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:05.968926 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:34:06.399955 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:34:06.399870 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:34:36.399830 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:34:36.399792 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:34:49.142748 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:49.142708 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:34:57.537215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:57.535307 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:34:57.537215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:34:57.535401 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:35:04.395655 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.395609 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" podUID="be173d02-3479-44b4-97b3-78142c757a80" containerName="main" containerID="cri-o://08ac002fc1bdff99431b164fa96ba1d75fb37ff3eeb5f4c993de4fefc9e4b6c8" gracePeriod=45 Nov 04 03:35:04.524946 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.524915 4784 generic.go:356] "Generic (PLEG): container finished" podID="be173d02-3479-44b4-97b3-78142c757a80" containerID="08ac002fc1bdff99431b164fa96ba1d75fb37ff3eeb5f4c993de4fefc9e4b6c8" exitCode=0 Nov 04 03:35:04.525106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.524984 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" event={"ID":"be173d02-3479-44b4-97b3-78142c757a80","Type":"ContainerDied","Data":"08ac002fc1bdff99431b164fa96ba1d75fb37ff3eeb5f4c993de4fefc9e4b6c8"} Nov 04 03:35:04.657411 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.657344 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:35:04.718675 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718636 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.718884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718688 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\" (UniqueName: \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.718884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718714 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.718884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718757 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.718884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718785 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.719126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718905 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\") pod \"be173d02-3479-44b4-97b3-78142c757a80\" (UID: \"be173d02-3479-44b4-97b3-78142c757a80\") " Nov 04 03:35:04.719126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.718955 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:04.719126 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.719108 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:04.719301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.719216 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:04.719340 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.719332 4784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-home\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:04.719376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.719347 4784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-tmp-dir\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:04.719376 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.719356 4784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-model-cache\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:04.721282 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.721263 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "kube-api-access-cmp6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:04.721570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.721544 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:04.721570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.721555 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs" pod "be173d02-3479-44b4-97b3-78142c757a80" (UID: "be173d02-3479-44b4-97b3-78142c757a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:35:04.820722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.820680 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmp6q\" (UniqueName: \"kubernetes.io/projected/be173d02-3479-44b4-97b3-78142c757a80-kube-api-access-cmp6q\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:04.820722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.820711 4784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be173d02-3479-44b4-97b3-78142c757a80-tls-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:04.820722 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:04.820719 4784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be173d02-3479-44b4-97b3-78142c757a80-dshm\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:05.532084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:05.531996 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" Nov 04 03:35:05.532084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:05.532010 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54" event={"ID":"be173d02-3479-44b4-97b3-78142c757a80","Type":"ContainerDied","Data":"d70701ffcdfda0eb6395d1182215d069711bdc8db7773d97c47b291484eee154"} Nov 04 03:35:05.532084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:05.532060 4784 scope.go:122] "RemoveContainer" containerID="08ac002fc1bdff99431b164fa96ba1d75fb37ff3eeb5f4c993de4fefc9e4b6c8" Nov 04 03:35:05.703100 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:05.703061 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:35:05.752501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:05.752460 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b05dd916-kserve-77766fbc7c-jbv54"] Nov 04 03:35:06.051948 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:06.051911 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks"] Nov 04 03:35:06.054469 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:06.054433 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2zmks"] Nov 04 03:35:06.400878 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:35:06.400789 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:35:07.162656 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:07.162623 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e824942-0c48-4404-a2ac-e93762472004" path="/var/lib/kubelet/pods/0e824942-0c48-4404-a2ac-e93762472004/volumes" Nov 04 03:35:07.163072 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:07.163035 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be173d02-3479-44b4-97b3-78142c757a80" path="/var/lib/kubelet/pods/be173d02-3479-44b4-97b3-78142c757a80/volumes" Nov 04 03:35:14.925819 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:14.925777 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:35:14.926489 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:14.926454 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:35:36.400326 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:35:36.400288 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:35:38.021059 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.021025 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:35:38.021583 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.021568 4784 cpu_manager.go:414] "containerMap: removing container" podUID="be173d02-3479-44b4-97b3-78142c757a80" containerName="main" Nov 04 03:35:38.021634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.021585 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be173d02-3479-44b4-97b3-78142c757a80" containerName="main" Nov 04 03:35:38.021679 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.021665 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="be173d02-3479-44b4-97b3-78142c757a80" containerName="main" Nov 04 03:35:38.025889 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.025872 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.079723 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.079684 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dffbe0c3-openshift-default-dockercfg-2l2z9\"" Nov 04 03:35:38.096459 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096418 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.096691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096479 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.096691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096533 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.096691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096591 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzggp\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.096691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096639 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.096691 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096687 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.097127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096712 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.097127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096750 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.097127 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.096799 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.134151 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.134116 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:35:38.198182 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198143 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198240 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzggp\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198276 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198308 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198328 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198405 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198352 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198415 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198471 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.198681 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198509 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.199084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198676 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.199084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198696 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.199084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.198876 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.199084 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.199026 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.199390 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.199274 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.201146 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.201128 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.201408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.201387 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.262775 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.262736 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.262941 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.262743 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzggp\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\") pod \"e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.339520 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.339433 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:38.770721 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.770683 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:35:38.771657 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:35:38.771635 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3b3613_eeec_4462_8573_9c0dcb54d3f3.slice/crio-35f28c780d02895ea72a3585a4043d8ac092147e92afc624f74e77373e743dcb WatchSource:0}: Error finding container 35f28c780d02895ea72a3585a4043d8ac092147e92afc624f74e77373e743dcb: Status 404 returned error can't find the container with id 35f28c780d02895ea72a3585a4043d8ac092147e92afc624f74e77373e743dcb Nov 04 03:35:38.773971 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.773933 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:35:38.774046 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.774003 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:35:38.774046 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.774034 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:35:38.791945 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:38.791836 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerStarted","Data":"35f28c780d02895ea72a3585a4043d8ac092147e92afc624f74e77373e743dcb"} Nov 04 03:35:39.800269 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:39.800225 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerStarted","Data":"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72"} Nov 04 03:35:40.024728 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:40.024670 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podStartSLOduration=3.024655358 podStartE2EDuration="3.024655358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:39.973761208 +0000 UTC m=+3786.940262203" watchObservedRunningTime="2026-11-04 03:35:40.024655358 +0000 UTC m=+3786.991156349" Nov 04 03:35:40.339883 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:40.339842 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:40.345280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:40.345247 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:40.805731 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:40.805690 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:35:40.806635 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:35:40.806618 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:03.944795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.944705 4784 generic.go:356] "Generic (PLEG): container finished" podID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerID="a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72" exitCode=0 Nov 04 03:36:03.945271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.944787 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerDied","Data":"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72"} Nov 04 03:36:03.945271 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.945236 4784 scope.go:122] "RemoveContainer" containerID="a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72" Nov 04 03:36:03.946513 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.946474 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:36:03.946603 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.946580 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:36:03.946653 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:03.946615 4784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 03:36:04.984527 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:04.984469 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerStarted","Data":"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914"} Nov 04 03:36:05.340342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:05.340252 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:05.340516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:05.340461 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 04 03:36:05.340562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:05.340517 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 04 03:36:06.340118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:06.340082 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 04 03:36:06.340588 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:06.340152 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 04 03:36:06.407129 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:36:06.407094 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:36:07.340624 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:07.340585 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 04 03:36:07.341042 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:07.340653 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 04 03:36:08.339813 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:08.339770 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 04 03:36:08.340098 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:08.339827 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 04 03:36:09.339830 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:09.339792 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 04 03:36:09.340295 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:09.339857 4784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 04 03:36:10.344567 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:10.344533 4784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:10.345082 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:10.344945 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:10.345832 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:10.345813 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:18.605608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.605515 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc"] Nov 04 03:36:18.609639 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.609613 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.666458 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.666423 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc"] Nov 04 03:36:18.681142 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.681097 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:36:18.681325 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.681150 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:36:18.714344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.714316 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4gxbl\"" Nov 04 03:36:18.714344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.714316 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:36:18.744344 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.744307 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.744549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.744389 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.744549 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.744496 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c29qv\" (UniqueName: \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.845092 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.845047 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.845338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.845119 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c29qv\" (UniqueName: \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.845338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.845219 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.845899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.845832 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.847744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.847719 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.907921 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.907846 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c29qv\" (UniqueName: \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\") pod \"maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:18.922666 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:18.922638 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:19.299669 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:36:19.299639 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b92dca7_c5f1_491e_871d_2a44288f4bcf.slice/crio-3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8 WatchSource:0}: Error finding container 3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8: Status 404 returned error can't find the container with id 3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8 Nov 04 03:36:19.300063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:19.300040 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc"] Nov 04 03:36:20.086496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:20.086454 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" event={"ID":"6b92dca7-c5f1-491e-871d-2a44288f4bcf","Type":"ContainerStarted","Data":"4237ec42ac03c153f897333b7cd1e3a2128708649f36805843f5337292602d26"} Nov 04 03:36:20.086914 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:20.086547 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" event={"ID":"6b92dca7-c5f1-491e-871d-2a44288f4bcf","Type":"ContainerStarted","Data":"3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8"} Nov 04 03:36:20.206129 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:20.206067 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" podStartSLOduration=2.206052307 podStartE2EDuration="2.206052307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:20.204635697 +0000 UTC m=+3827.171136689" watchObservedRunningTime="2026-11-04 03:36:20.206052307 +0000 UTC m=+3827.172553299" Nov 04 03:36:21.094640 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:21.094605 4784 generic.go:356] "Generic (PLEG): container finished" podID="6b92dca7-c5f1-491e-871d-2a44288f4bcf" containerID="4237ec42ac03c153f897333b7cd1e3a2128708649f36805843f5337292602d26" exitCode=0 Nov 04 03:36:21.095081 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:21.094683 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" event={"ID":"6b92dca7-c5f1-491e-871d-2a44288f4bcf","Type":"ContainerDied","Data":"4237ec42ac03c153f897333b7cd1e3a2128708649f36805843f5337292602d26"} Nov 04 03:36:22.246199 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.246160 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:22.379793 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.379700 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\" (UniqueName: \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\") pod \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " Nov 04 03:36:22.379793 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.379765 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\") pod \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " Nov 04 03:36:22.380057 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.379834 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\") pod \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\" (UID: \"6b92dca7-c5f1-491e-871d-2a44288f4bcf\") " Nov 04 03:36:22.380279 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.380254 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca" pod "6b92dca7-c5f1-491e-871d-2a44288f4bcf" (UID: "6b92dca7-c5f1-491e-871d-2a44288f4bcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:22.382113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.382082 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv" pod "6b92dca7-c5f1-491e-871d-2a44288f4bcf" (UID: "6b92dca7-c5f1-491e-871d-2a44288f4bcf"). InnerVolumeSpecName "kube-api-access-c29qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:22.382674 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.382647 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca" pod "6b92dca7-c5f1-491e-871d-2a44288f4bcf" (UID: "6b92dca7-c5f1-491e-871d-2a44288f4bcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:36:22.480852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.480806 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c29qv\" (UniqueName: \"kubernetes.io/projected/6b92dca7-c5f1-491e-871d-2a44288f4bcf-kube-api-access-c29qv\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:22.480852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.480845 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:22.480852 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:22.480856 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b92dca7-c5f1-491e-871d-2a44288f4bcf-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:23.107377 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:23.107344 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" Nov 04 03:36:23.107576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:23.107384 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc" event={"ID":"6b92dca7-c5f1-491e-871d-2a44288f4bcf","Type":"ContainerDied","Data":"3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8"} Nov 04 03:36:23.107576 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:23.107442 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3532077b1fbe8648b95e6c7ccfde7877c92f037edb176ab4913cb3824b38d8e8" Nov 04 03:36:36.400489 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:36:36.400448 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:36:37.148760 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:37.148720 4784 scope.go:122] "RemoveContainer" containerID="8f79c457e3e67f0e4fdcecafac0f724ad24a862305dbea6ffef78bf4ef5f98d8" Nov 04 03:36:38.451492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:38.451455 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc"] Nov 04 03:36:38.502425 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:38.502388 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dffbe0c3-p8dtc"] Nov 04 03:36:39.159742 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:39.159698 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b92dca7-c5f1-491e-871d-2a44288f4bcf" path="/var/lib/kubelet/pods/6b92dca7-c5f1-491e-871d-2a44288f4bcf/volumes" Nov 04 03:36:39.624956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:39.624912 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:36:39.625489 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:39.625269 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" containerID="cri-o://3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914" gracePeriod=30 Nov 04 03:36:40.347618 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:40.347584 4784 patch_prober.go:29] interesting pod/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:36:40.347820 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:40.347640 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:36:44.874248 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.874220 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:44.909688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909617 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909656 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909688 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909681 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909710 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909755 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909795 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909842 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909871 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909911 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:44.909955 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.909920 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\") pod \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\" (UID: \"8c3b3613-eeec-4462-8573-9c0dcb54d3f3\") " Nov 04 03:36:44.910366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910135 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:44.910366 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910172 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:44.910487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910306 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:44.910487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910377 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:44.910487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910460 4784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:44.910487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910475 4784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-credential-socket\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:44.910487 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.910485 4784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-data\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:44.912277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.912211 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "kube-api-access-lzggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:44.912277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.912220 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:44.912457 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.912351 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:44.912666 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:44.912649 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo" pod "8c3b3613-eeec-4462-8573-9c0dcb54d3f3" (UID: "8c3b3613-eeec-4462-8573-9c0dcb54d3f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:36:45.010997 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.010964 4784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-podinfo\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.010997 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.010993 4784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-envoy\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.010997 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.011004 4784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-workload-certs\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.011285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.011014 4784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istiod-ca-cert\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.011285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.011023 4784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-istio-token\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.011285 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.011032 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzggp\" (UniqueName: \"kubernetes.io/projected/8c3b3613-eeec-4462-8573-9c0dcb54d3f3-kube-api-access-lzggp\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:45.237159 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.237119 4784 generic.go:356] "Generic (PLEG): container finished" podID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerID="3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914" exitCode=0 Nov 04 03:36:45.237391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.237228 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" Nov 04 03:36:45.237391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.237235 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerDied","Data":"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914"} Nov 04 03:36:45.237391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.237329 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z" event={"ID":"8c3b3613-eeec-4462-8573-9c0dcb54d3f3","Type":"ContainerDied","Data":"35f28c780d02895ea72a3585a4043d8ac092147e92afc624f74e77373e743dcb"} Nov 04 03:36:45.237391 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.237359 4784 scope.go:122] "RemoveContainer" containerID="3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914" Nov 04 03:36:45.247241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.247214 4784 scope.go:122] "RemoveContainer" containerID="a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72" Nov 04 03:36:45.256511 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.256485 4784 scope.go:122] "RemoveContainer" containerID="3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914" Nov 04 03:36:45.256828 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:36:45.256806 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914\": container with ID starting with 3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914 not found: ID does not exist" containerID="3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914" Nov 04 03:36:45.256897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.256840 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914"} err="failed to get container status \"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914\": rpc error: code = NotFound desc = could not find container \"3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914\": container with ID starting with 3c130253899b2554033d5baabc76740781a17835054c9db2c04f3f7d3db14914 not found: ID does not exist" Nov 04 03:36:45.256897 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.256860 4784 scope.go:122] "RemoveContainer" containerID="a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72" Nov 04 03:36:45.257151 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:36:45.257128 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72\": container with ID starting with a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72 not found: ID does not exist" containerID="a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72" Nov 04 03:36:45.257264 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.257162 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72"} err="failed to get container status \"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72\": rpc error: code = NotFound desc = could not find container \"a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72\": container with ID starting with a40d6337e20b803f9fb60bac6f2e7dd553714e3c50d626c22c3c86f38fb58c72 not found: ID does not exist" Nov 04 03:36:45.408246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.408183 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:36:45.412224 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:45.412198 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dffbe0c3-openshift-default-857cd74dd7-5sn8z"] Nov 04 03:36:47.159485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:47.159451 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" path="/var/lib/kubelet/pods/8c3b3613-eeec-4462-8573-9c0dcb54d3f3/volumes" Nov 04 03:36:49.933704 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.933661 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:36:49.934162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934116 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934128 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934138 4784 cpu_manager.go:414] "containerMap: removing container" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934162 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934143 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934168 4784 cpu_manager.go:414] "containerMap: removing container" podUID="6b92dca7-c5f1-491e-871d-2a44288f4bcf" containerName="revoke-keys" Nov 04 03:36:49.934381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934173 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b92dca7-c5f1-491e-871d-2a44288f4bcf" containerName="revoke-keys" Nov 04 03:36:49.934381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934319 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934331 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3b3613-eeec-4462-8573-9c0dcb54d3f3" containerName="istio-proxy" Nov 04 03:36:49.934381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.934341 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b92dca7-c5f1-491e-871d-2a44288f4bcf" containerName="revoke-keys" Nov 04 03:36:49.940307 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.940273 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:49.964360 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.964325 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vffg\" (UniqueName: \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:49.964548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.964413 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:49.996309 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:49.996274 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:36:50.044122 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.044089 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-cc2474\"" Nov 04 03:36:50.044367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.044237 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-cc2474-dockercfg-sdwp2\"" Nov 04 03:36:50.065734 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.065699 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:50.065925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.065780 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vffg\" (UniqueName: \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:50.066437 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.066419 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:50.129375 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.129345 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vffg\" (UniqueName: \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\") pod \"payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:50.254854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.254761 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:50.637305 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:50.637269 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:36:50.638974 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:36:50.638948 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e602da_317b_45fb_a917_b41f4ce38e6d.slice/crio-7a94897970600e30b298b195d32763c25adf340beb0943955a86467ab8b9203e WatchSource:0}: Error finding container 7a94897970600e30b298b195d32763c25adf340beb0943955a86467ab8b9203e: Status 404 returned error can't find the container with id 7a94897970600e30b298b195d32763c25adf340beb0943955a86467ab8b9203e Nov 04 03:36:51.274268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:51.274227 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" event={"ID":"53e602da-317b-45fb-a917-b41f4ce38e6d","Type":"ContainerStarted","Data":"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91"} Nov 04 03:36:51.274705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:51.274289 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" event={"ID":"53e602da-317b-45fb-a917-b41f4ce38e6d","Type":"ContainerStarted","Data":"7a94897970600e30b298b195d32763c25adf340beb0943955a86467ab8b9203e"} Nov 04 03:36:51.274705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:51.274317 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:36:51.395033 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:51.394980 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" podStartSLOduration=2.394966721 podStartE2EDuration="2.394966721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:51.392119979 +0000 UTC m=+3858.358620973" watchObservedRunningTime="2026-11-04 03:36:51.394966721 +0000 UTC m=+3858.361467712" Nov 04 03:36:59.050132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:59.050095 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97"] Nov 04 03:36:59.100796 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:59.100759 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecfbe564-mhv97"] Nov 04 03:36:59.159994 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:36:59.159963 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9e63d9-c187-4668-8fe4-dbb526ccbe92" path="/var/lib/kubelet/pods/4d9e63d9-c187-4668-8fe4-dbb526ccbe92/volumes" Nov 04 03:37:02.283206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:37:02.283151 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:37:06.400213 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:37:06.400159 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:37:34.417992 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:37:34.417963 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:37:34.423501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:37:34.423477 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:37:34.513755 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:37:34.513724 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:37:34.518967 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:37:34.518946 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:37:36.400379 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:37:36.400335 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:38:06.400727 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:38:06.400685 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:38:36.400052 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:38:36.400007 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:38:37.192528 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:37.192495 4784 scope.go:122] "RemoveContainer" containerID="02b0bd68224a49f5109defccc1b1b065b09494bf4c78271562e95e578b5c7539" Nov 04 03:38:38.293853 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:38.293817 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:38:43.295152 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.295110 4784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" podUID="53e602da-317b-45fb-a917-b41f4ce38e6d" containerName="payload-processing" containerID="cri-o://9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91" gracePeriod=25 Nov 04 03:38:43.571797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.571768 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:38:43.655289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.655255 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\") pod \"53e602da-317b-45fb-a917-b41f4ce38e6d\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " Nov 04 03:38:43.655509 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.655352 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\" (UniqueName: \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\") pod \"53e602da-317b-45fb-a917-b41f4ce38e6d\" (UID: \"53e602da-317b-45fb-a917-b41f4ce38e6d\") " Nov 04 03:38:43.655736 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.655709 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume" pod "53e602da-317b-45fb-a917-b41f4ce38e6d" (UID: "53e602da-317b-45fb-a917-b41f4ce38e6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:43.657811 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.657782 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg" pod "53e602da-317b-45fb-a917-b41f4ce38e6d" (UID: "53e602da-317b-45fb-a917-b41f4ce38e6d"). InnerVolumeSpecName "kube-api-access-6vffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:43.756428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.756387 4784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e602da-317b-45fb-a917-b41f4ce38e6d-plugins-config-volume\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:43.756428 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.756422 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vffg\" (UniqueName: \"kubernetes.io/projected/53e602da-317b-45fb-a917-b41f4ce38e6d-kube-api-access-6vffg\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:43.948701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.948666 4784 generic.go:356] "Generic (PLEG): container finished" podID="53e602da-317b-45fb-a917-b41f4ce38e6d" containerID="9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91" exitCode=0 Nov 04 03:38:43.948905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.948718 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" Nov 04 03:38:43.948905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.948754 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" event={"ID":"53e602da-317b-45fb-a917-b41f4ce38e6d","Type":"ContainerDied","Data":"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91"} Nov 04 03:38:43.948905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.948810 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc" event={"ID":"53e602da-317b-45fb-a917-b41f4ce38e6d","Type":"ContainerDied","Data":"7a94897970600e30b298b195d32763c25adf340beb0943955a86467ab8b9203e"} Nov 04 03:38:43.948905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.948835 4784 scope.go:122] "RemoveContainer" containerID="9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91" Nov 04 03:38:43.958834 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.958813 4784 scope.go:122] "RemoveContainer" containerID="9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91" Nov 04 03:38:43.959111 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:38:43.959091 4784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91\": container with ID starting with 9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91 not found: ID does not exist" containerID="9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91" Nov 04 03:38:43.959160 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:43.959121 4784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91"} err="failed to get container status \"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91\": rpc error: code = NotFound desc = could not find container \"9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91\": container with ID starting with 9a9770c4e97de35788bb93adc022734e1a6f5540f26df1a8fd230e2cbc8c8b91 not found: ID does not exist" Nov 04 03:38:44.117643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:44.117605 4784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:38:44.121904 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:44.121877 4784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-cc2474-684cc5b57d-fzbbc"] Nov 04 03:38:45.159909 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:45.159870 4784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e602da-317b-45fb-a917-b41f4ce38e6d" path="/var/lib/kubelet/pods/53e602da-317b-45fb-a917-b41f4ce38e6d/volumes" Nov 04 03:38:57.542922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:57.542384 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v59qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:38:57.542922 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:38:57.542460 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v59qw" podUID="833f3303-97e5-4a75-9a2e-05a4b2aac450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:39:06.558766 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:39:06.558717 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:39:11.198647 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.198606 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9"] Nov 04 03:39:11.199085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.199066 4784 cpu_manager.go:414] "containerMap: removing container" podUID="53e602da-317b-45fb-a917-b41f4ce38e6d" containerName="payload-processing" Nov 04 03:39:11.199085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.199077 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e602da-317b-45fb-a917-b41f4ce38e6d" containerName="payload-processing" Nov 04 03:39:11.199239 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.199157 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e602da-317b-45fb-a917-b41f4ce38e6d" containerName="payload-processing" Nov 04 03:39:11.202406 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.202389 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.260334 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.260298 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9"] Nov 04 03:39:11.308148 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.308108 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:39:11.308362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.308161 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:39:11.308362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.308165 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4gxbl\"" Nov 04 03:39:11.308362 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.308212 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:39:11.365634 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.365595 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.365826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.365706 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klq52\" (UniqueName: \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.365826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.365761 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.466643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.466549 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.466643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.466624 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klq52\" (UniqueName: \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.466859 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.466664 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.467253 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.467223 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.469225 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.469174 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.529658 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.529620 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klq52\" (UniqueName: \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\") pod \"maas-api-revoke-sub-f03629db-h9zg9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:11.814666 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:11.814579 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:12.194408 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:12.193615 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9"] Nov 04 03:39:12.239885 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:12.239846 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" event={"ID":"d2ef5df4-41ff-42da-8559-98ed84d15fc9","Type":"ContainerStarted","Data":"d585f4cb563f0976351cccc477b170173be3fb3fb7064fce677a9de1ca450b14"} Nov 04 03:39:13.247574 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:13.247537 4784 generic.go:356] "Generic (PLEG): container finished" podID="d2ef5df4-41ff-42da-8559-98ed84d15fc9" containerID="7c52f972371c10cf1541b5230c46a831f0742c6c3d2b99d811d826995aca2f0e" exitCode=0 Nov 04 03:39:13.248049 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:13.247622 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" event={"ID":"d2ef5df4-41ff-42da-8559-98ed84d15fc9","Type":"ContainerDied","Data":"7c52f972371c10cf1541b5230c46a831f0742c6c3d2b99d811d826995aca2f0e"} Nov 04 03:39:14.405653 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.405623 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:14.497645 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.497603 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\") pod \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " Nov 04 03:39:14.497923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.497742 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\") pod \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " Nov 04 03:39:14.497923 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.497849 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\" (UniqueName: \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\") pod \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\" (UID: \"d2ef5df4-41ff-42da-8559-98ed84d15fc9\") " Nov 04 03:39:14.498099 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.498073 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca" pod "d2ef5df4-41ff-42da-8559-98ed84d15fc9" (UID: "d2ef5df4-41ff-42da-8559-98ed84d15fc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:14.498268 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.498253 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:14.500105 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.500083 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca" pod "d2ef5df4-41ff-42da-8559-98ed84d15fc9" (UID: "d2ef5df4-41ff-42da-8559-98ed84d15fc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:14.500204 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.500099 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52" pod "d2ef5df4-41ff-42da-8559-98ed84d15fc9" (UID: "d2ef5df4-41ff-42da-8559-98ed84d15fc9"). InnerVolumeSpecName "kube-api-access-klq52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:14.599797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.599757 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klq52\" (UniqueName: \"kubernetes.io/projected/d2ef5df4-41ff-42da-8559-98ed84d15fc9-kube-api-access-klq52\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:14.599797 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:14.599793 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ef5df4-41ff-42da-8559-98ed84d15fc9-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:15.262381 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:15.262349 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" Nov 04 03:39:15.262578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:15.262382 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h9zg9" event={"ID":"d2ef5df4-41ff-42da-8559-98ed84d15fc9","Type":"ContainerDied","Data":"d585f4cb563f0976351cccc477b170173be3fb3fb7064fce677a9de1ca450b14"} Nov 04 03:39:15.262578 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:15.262428 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d585f4cb563f0976351cccc477b170173be3fb3fb7064fce677a9de1ca450b14" Nov 04 03:39:29.929214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:29.928295 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-bdgtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:39:29.929214 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:29.928380 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-bdgtb" podUID="83ddc4ac-b046-42be-abf4-d5760f81e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:39:30.929471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:30.929330 4784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-j8cqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:39:30.929471 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:30.929460 4784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-j8cqc" podUID="3813f6c6-ecdf-4990-9be2-fb579378bf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:39:31.554725 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.554688 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj"] Nov 04 03:39:31.555232 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.555217 4784 cpu_manager.go:414] "containerMap: removing container" podUID="d2ef5df4-41ff-42da-8559-98ed84d15fc9" containerName="revoke-keys" Nov 04 03:39:31.555284 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.555234 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ef5df4-41ff-42da-8559-98ed84d15fc9" containerName="revoke-keys" Nov 04 03:39:31.555356 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.555346 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ef5df4-41ff-42da-8559-98ed84d15fc9" containerName="revoke-keys" Nov 04 03:39:31.558701 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.558682 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.664364 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.664327 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:39:31.664632 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.664337 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:39:31.664770 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.664395 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4gxbl\"" Nov 04 03:39:31.664931 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.664910 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:39:31.667492 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.667468 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj"] Nov 04 03:39:31.693094 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.693053 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.693324 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.693119 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.693324 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.693246 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mn59\" (UniqueName: \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.794507 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.794465 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.794741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.794527 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mn59\" (UniqueName: \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.794741 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.794611 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.795305 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.795285 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.797261 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.797236 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.857250 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.857126 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mn59\" (UniqueName: \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\") pod \"maas-api-revoke-sub-0165d7d2-fqrbj\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:31.872829 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:31.872793 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:32.247663 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:32.247633 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj"] Nov 04 03:39:32.248704 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:39:32.248677 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15581866_8a51_4c74_8db2_927659c1a75a.slice/crio-00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc WatchSource:0}: Error finding container 00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc: Status 404 returned error can't find the container with id 00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc Nov 04 03:39:32.412672 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:32.412639 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" event={"ID":"15581866-8a51-4c74-8db2-927659c1a75a","Type":"ContainerStarted","Data":"00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc"} Nov 04 03:39:33.420132 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:33.420097 4784 generic.go:356] "Generic (PLEG): container finished" podID="15581866-8a51-4c74-8db2-927659c1a75a" containerID="3344255838f833e6fd5da3e62a9fad5e3e921a5d848e3c2c23516435c9beb8b1" exitCode=0 Nov 04 03:39:33.420568 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:33.420152 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" event={"ID":"15581866-8a51-4c74-8db2-927659c1a75a","Type":"ContainerDied","Data":"3344255838f833e6fd5da3e62a9fad5e3e921a5d848e3c2c23516435c9beb8b1"} Nov 04 03:39:34.572643 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.572617 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:34.618085 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.618048 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\") pod \"15581866-8a51-4c74-8db2-927659c1a75a\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " Nov 04 03:39:34.618289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.618133 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\" (UniqueName: \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\") pod \"15581866-8a51-4c74-8db2-927659c1a75a\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " Nov 04 03:39:34.618289 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.618154 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\") pod \"15581866-8a51-4c74-8db2-927659c1a75a\" (UID: \"15581866-8a51-4c74-8db2-927659c1a75a\") " Nov 04 03:39:34.618581 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.618557 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca" pod "15581866-8a51-4c74-8db2-927659c1a75a" (UID: "15581866-8a51-4c74-8db2-927659c1a75a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:34.620392 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.620370 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca" pod "15581866-8a51-4c74-8db2-927659c1a75a" (UID: "15581866-8a51-4c74-8db2-927659c1a75a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:34.620496 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.620421 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59" pod "15581866-8a51-4c74-8db2-927659c1a75a" (UID: "15581866-8a51-4c74-8db2-927659c1a75a"). InnerVolumeSpecName "kube-api-access-8mn59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:34.718956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.718865 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:34.718956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.718900 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mn59\" (UniqueName: \"kubernetes.io/projected/15581866-8a51-4c74-8db2-927659c1a75a-kube-api-access-8mn59\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:34.718956 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:34.718910 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15581866-8a51-4c74-8db2-927659c1a75a-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:35.439635 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:35.439543 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" event={"ID":"15581866-8a51-4c74-8db2-927659c1a75a","Type":"ContainerDied","Data":"00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc"} Nov 04 03:39:35.439635 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:35.439581 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fqrbj" Nov 04 03:39:35.439635 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:35.439599 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00fa9f5b7e1bc1ffaabc444b2a15d5f69c4bc02af9f31080a5dcb2c6a75accbc" Nov 04 03:39:36.399639 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:39:36.399596 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:39:37.217757 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:37.217727 4784 scope.go:122] "RemoveContainer" containerID="ce2e3d1c93a591d328daac5ce205329669dbfeb56283f6fdd8f88c600065a5e8" Nov 04 03:39:43.333771 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.333730 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w"] Nov 04 03:39:43.334449 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.334431 4784 cpu_manager.go:414] "containerMap: removing container" podUID="15581866-8a51-4c74-8db2-927659c1a75a" containerName="revoke-keys" Nov 04 03:39:43.334499 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.334452 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15581866-8a51-4c74-8db2-927659c1a75a" containerName="revoke-keys" Nov 04 03:39:43.334608 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.334597 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="15581866-8a51-4c74-8db2-927659c1a75a" containerName="revoke-keys" Nov 04 03:39:43.338220 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.338202 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.394548 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.394514 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w"] Nov 04 03:39:43.406341 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.406304 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qshj\" (UniqueName: \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.406555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.406362 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.406555 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.406492 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.441918 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.441875 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:39:43.442128 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.441940 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:39:43.442128 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.441969 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:39:43.492064 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.492037 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4gxbl\"" Nov 04 03:39:43.507526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.507490 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qshj\" (UniqueName: \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.507526 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.507527 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.507779 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.507582 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.508319 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.508299 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.510052 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.510034 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.570442 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.570413 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qshj\" (UniqueName: \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\") pod \"maas-api-revoke-sub-6b19338f-9zd2w\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:43.652795 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:43.652712 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:44.029117 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:44.029088 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w"] Nov 04 03:39:44.030548 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:39:44.030521 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda493d673_42aa_4a30_9789_71fdca94dfeb.slice/crio-a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d WatchSource:0}: Error finding container a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d: Status 404 returned error can't find the container with id a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d Nov 04 03:39:44.496569 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:44.496531 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" event={"ID":"a493d673-42aa-4a30-9789-71fdca94dfeb","Type":"ContainerStarted","Data":"8b7053a317423fad6a579a39096f3965d26ba224c47b983ece4152b523f12ae9"} Nov 04 03:39:44.496964 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:44.496587 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" event={"ID":"a493d673-42aa-4a30-9789-71fdca94dfeb","Type":"ContainerStarted","Data":"a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d"} Nov 04 03:39:44.615504 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:44.615444 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" podStartSLOduration=1.615430807 podStartE2EDuration="1.615430807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:39:44.614565434 +0000 UTC m=+4031.581066427" watchObservedRunningTime="2026-11-04 03:39:44.615430807 +0000 UTC m=+4031.581931777" Nov 04 03:39:45.504355 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:45.504320 4784 generic.go:356] "Generic (PLEG): container finished" podID="a493d673-42aa-4a30-9789-71fdca94dfeb" containerID="8b7053a317423fad6a579a39096f3965d26ba224c47b983ece4152b523f12ae9" exitCode=0 Nov 04 03:39:45.504789 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:45.504411 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" event={"ID":"a493d673-42aa-4a30-9789-71fdca94dfeb","Type":"ContainerDied","Data":"8b7053a317423fad6a579a39096f3965d26ba224c47b983ece4152b523f12ae9"} Nov 04 03:39:46.649121 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.649093 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:46.739025 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.738992 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\" (UniqueName: \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\") pod \"a493d673-42aa-4a30-9789-71fdca94dfeb\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " Nov 04 03:39:46.739251 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.739111 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\") pod \"a493d673-42aa-4a30-9789-71fdca94dfeb\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " Nov 04 03:39:46.739251 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.739141 4784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\") pod \"a493d673-42aa-4a30-9789-71fdca94dfeb\" (UID: \"a493d673-42aa-4a30-9789-71fdca94dfeb\") " Nov 04 03:39:46.739628 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.739593 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca" pod "a493d673-42aa-4a30-9789-71fdca94dfeb" (UID: "a493d673-42aa-4a30-9789-71fdca94dfeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:46.741310 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.741285 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca" pod "a493d673-42aa-4a30-9789-71fdca94dfeb" (UID: "a493d673-42aa-4a30-9789-71fdca94dfeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:46.741414 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.741349 4784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj" pod "a493d673-42aa-4a30-9789-71fdca94dfeb" (UID: "a493d673-42aa-4a30-9789-71fdca94dfeb"). InnerVolumeSpecName "kube-api-access-5qshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:46.840472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.840377 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-xks-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:46.840472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.840410 4784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a493d673-42aa-4a30-9789-71fdca94dfeb-maas-api-openshift-ca\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:46.840472 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:46.840423 4784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qshj\" (UniqueName: \"kubernetes.io/projected/a493d673-42aa-4a30-9789-71fdca94dfeb-kube-api-access-5qshj\") on node \"ip-10-0-133-209.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:47.517358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:47.517276 4784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" Nov 04 03:39:47.517358 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:47.517313 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9zd2w" event={"ID":"a493d673-42aa-4a30-9789-71fdca94dfeb","Type":"ContainerDied","Data":"a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d"} Nov 04 03:39:47.517565 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:47.517361 4784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a834e65975923f531a10ce6e73a51d39ffd3b3e264a813a5a06e6e86dc84699d" Nov 04 03:39:55.727063 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:39:55.727025 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-94c54b89b-7zxv2_100c05c7-eb70-4e04-93a4-e405b1f2c0fa/authorino/0.log" Nov 04 03:40:06.400300 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:40:06.400250 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:40:08.016246 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:08.016214 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8b7c7f7bc-xh4z4_79c7a142-0ca3-4ed5-bfc5-7e75694180af/manager/0.log" Nov 04 03:40:08.669163 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:08.669130 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-k7c2j_e830410b-0936-4d99-9932-ffa515c1760a/manager/0.log" Nov 04 03:40:09.395206 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:09.395162 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:40:09.988884 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:09.988848 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dbq49_c4d067ed-305b-40ee-bb99-3c3cab436dab/manager/0.log" Nov 04 03:40:12.129972 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:12.129939 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fqrbj_15581866-8a51-4c74-8db2-927659c1a75a/revoke-keys/0.log" Nov 04 03:40:12.358656 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:12.358625 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9zd2w_a493d673-42aa-4a30-9789-71fdca94dfeb/revoke-keys/0.log" Nov 04 03:40:12.804375 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:12.804344 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-h9zg9_d2ef5df4-41ff-42da-8559-98ed84d15fc9/revoke-keys/0.log" Nov 04 03:40:15.009342 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:15.009316 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-94c54b89b-7zxv2_100c05c7-eb70-4e04-93a4-e405b1f2c0fa/authorino/0.log" Nov 04 03:40:15.825436 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:15.825406 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9vc9j_b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a/manager/0.log" Nov 04 03:40:17.225014 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:17.224977 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-68mzs_792ef3b5-5678-4de5-9764-86a3b17138c8/manager/0.log" Nov 04 03:40:18.423632 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:18.423599 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b4848996xtj_f832f885-5e72-4892-8425-95fd377a536a/istio-proxy/0.log" Nov 04 03:40:19.953202 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:19.953130 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-bdgtb_83ddc4ac-b046-42be-abf4-d5760f81e9d0/istio-proxy/0.log" Nov 04 03:40:20.714280 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:20.714230 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-j8cqc_3813f6c6-ecdf-4990-9be2-fb579378bf66/istio-proxy/0.log" Nov 04 03:40:21.001998 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:21.001912 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-jwwww_f11de3b8-689f-4e98-b418-0a1794cf7907/istio-proxy/0.log" Nov 04 03:40:21.520651 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:21.520614 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-v59qw_833f3303-97e5-4a75-9a2e-05a4b2aac450/istio-proxy/0.log" Nov 04 03:40:22.014451 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:22.014423 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2_08528d88-4550-4f16-8e49-24e22757f3d7/istio-proxy/0.log" Nov 04 03:40:22.244034 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:22.243999 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-r82l2_55946bcd-f633-4787-ac61-ce256b606eea/payload-pre-processing/0.log" Nov 04 03:40:22.476106 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:22.476047 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:40:22.711113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:22.711058 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76598c4567-mhntk_f7e78e94-cc94-496f-ac33-f4e708709d9a/router/0.log" Nov 04 03:40:23.827241 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:23.827210 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc_9322be64-e971-47c8-aa8c-7b513cd2d4d9/main/0.log" Nov 04 03:40:24.117394 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:24.117291 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rfdbd_8e451643-d5cb-4de9-9b6e-f4d7ce68e625/main/0.log" Nov 04 03:40:24.886676 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:24.886645 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f_195055f3-cf50-4480-b49b-e185904110d8/main/0.log" Nov 04 03:40:25.435367 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:25.435340 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw_27aed879-c829-4755-98ea-9d12676f9bf5/main/0.log" Nov 04 03:40:26.980925 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:26.980888 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-94c54b89b-7zxv2_100c05c7-eb70-4e04-93a4-e405b1f2c0fa/authorino/0.log" Nov 04 03:40:36.400037 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:40:36.399994 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:40:36.481249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:36.481212 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8b7c7f7bc-xh4z4_79c7a142-0ca3-4ed5-bfc5-7e75694180af/manager/0.log" Nov 04 03:40:37.141501 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:37.141470 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-k7c2j_e830410b-0936-4d99-9932-ffa515c1760a/manager/0.log" Nov 04 03:40:37.808670 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:37.808627 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:40:38.391406 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:38.391369 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dbq49_c4d067ed-305b-40ee-bb99-3c3cab436dab/manager/0.log" Nov 04 03:40:39.843395 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:39.843367 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fqrbj_15581866-8a51-4c74-8db2-927659c1a75a/revoke-keys/0.log" Nov 04 03:40:40.063296 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:40.063266 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9zd2w_a493d673-42aa-4a30-9789-71fdca94dfeb/revoke-keys/0.log" Nov 04 03:40:40.289533 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:40.289501 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-h9zg9_d2ef5df4-41ff-42da-8559-98ed84d15fc9/revoke-keys/0.log" Nov 04 03:40:41.697926 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:41.697863 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-94c54b89b-7zxv2_100c05c7-eb70-4e04-93a4-e405b1f2c0fa/authorino/0.log" Nov 04 03:40:42.493413 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:42.493385 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9vc9j_b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a/manager/0.log" Nov 04 03:40:43.664161 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:43.664127 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-68mzs_792ef3b5-5678-4de5-9764-86a3b17138c8/manager/0.log" Nov 04 03:40:44.419558 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:44.419525 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b4848996xtj_f832f885-5e72-4892-8425-95fd377a536a/istio-proxy/0.log" Nov 04 03:40:45.866705 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:45.866670 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-bdgtb_83ddc4ac-b046-42be-abf4-d5760f81e9d0/istio-proxy/0.log" Nov 04 03:40:46.626987 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:46.626956 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-j8cqc_3813f6c6-ecdf-4990-9be2-fb579378bf66/istio-proxy/0.log" Nov 04 03:40:46.949249 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:46.949219 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-jwwww_f11de3b8-689f-4e98-b418-0a1794cf7907/istio-proxy/0.log" Nov 04 03:40:47.549485 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:47.549457 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-v59qw_833f3303-97e5-4a75-9a2e-05a4b2aac450/istio-proxy/0.log" Nov 04 03:40:48.108530 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:48.108495 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-w62d2_08528d88-4550-4f16-8e49-24e22757f3d7/istio-proxy/0.log" Nov 04 03:40:48.394169 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:48.394083 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-r82l2_55946bcd-f633-4787-ac61-ce256b606eea/payload-pre-processing/0.log" Nov 04 03:40:48.625337 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:48.625309 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:40:48.883029 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:48.882995 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76598c4567-mhntk_f7e78e94-cc94-496f-ac33-f4e708709d9a/router/0.log" Nov 04 03:40:49.510685 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:49.510653 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8dpvc_9322be64-e971-47c8-aa8c-7b513cd2d4d9/main/0.log" Nov 04 03:40:49.804943 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:49.804853 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rfdbd_8e451643-d5cb-4de9-9b6e-f4d7ce68e625/main/0.log" Nov 04 03:40:50.677277 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:50.677248 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88jv6f_195055f3-cf50-4480-b49b-e185904110d8/main/0.log" Nov 04 03:40:51.261902 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:40:51.261868 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2nsfw_27aed879-c829-4755-98ea-9d12676f9bf5/main/0.log" Nov 04 03:41:06.400398 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:41:06.400360 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:41:18.999715 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:18.999681 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:41:19.533827 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:19.533788 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:41:20.292976 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:20.292947 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:41:20.510577 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:20.510547 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:41:36.400318 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:41:36.400277 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:41:54.666975 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:54.666916 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:41:55.195422 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:55.195367 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78787b6955-hskxr_e82a765d-f8dc-428e-bce3-7d35821d8944/manager/0.log" Nov 04 03:41:55.983054 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:55.983024 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:41:56.209288 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.209261 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bqh9n_dad5fb32-3e75-44b7-8e86-6498bc5af637/payload-processing/0.log" Nov 04 03:41:56.575570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.575532 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjk9m/must-gather-j8mkj"] Nov 04 03:41:56.576010 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.575998 4784 cpu_manager.go:414] "containerMap: removing container" podUID="a493d673-42aa-4a30-9789-71fdca94dfeb" containerName="revoke-keys" Nov 04 03:41:56.576062 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.576011 4784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a493d673-42aa-4a30-9789-71fdca94dfeb" containerName="revoke-keys" Nov 04 03:41:56.576121 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.576112 4784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a493d673-42aa-4a30-9789-71fdca94dfeb" containerName="revoke-keys" Nov 04 03:41:56.579602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.579584 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.633836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.633803 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjk9m\"/\"openshift-service-ca.crt\"" Nov 04 03:41:56.633836 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.633831 4784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjk9m\"/\"kube-root-ca.crt\"" Nov 04 03:41:56.684854 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.684823 4784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjk9m\"/\"default-dockercfg-fdwhv\"" Nov 04 03:41:56.724964 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.724931 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4bz\" (UniqueName: \"kubernetes.io/projected/449df0b7-c235-403b-bda3-991d5b85e355-kube-api-access-9n4bz\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.725141 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.724985 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/449df0b7-c235-403b-bda3-991d5b85e355-must-gather-output\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.794744 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.794712 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/must-gather-j8mkj"] Nov 04 03:41:56.826497 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.826421 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/449df0b7-c235-403b-bda3-991d5b85e355-must-gather-output\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.826648 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.826530 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4bz\" (UniqueName: \"kubernetes.io/projected/449df0b7-c235-403b-bda3-991d5b85e355-kube-api-access-9n4bz\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.826774 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.826748 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/449df0b7-c235-403b-bda3-991d5b85e355-must-gather-output\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.888572 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.888542 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4bz\" (UniqueName: \"kubernetes.io/projected/449df0b7-c235-403b-bda3-991d5b85e355-kube-api-access-9n4bz\") pod \"must-gather-j8mkj\" (UID: \"449df0b7-c235-403b-bda3-991d5b85e355\") " pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:56.894515 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:56.894492 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" Nov 04 03:41:57.270118 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:57.270084 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/must-gather-j8mkj"] Nov 04 03:41:57.271565 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:41:57.271537 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod449df0b7_c235_403b_bda3_991d5b85e355.slice/crio-01bcbfafd092ad8fedeb9c7bdf08c9cedca07fb80b004317e4451ecdea02304e WatchSource:0}: Error finding container 01bcbfafd092ad8fedeb9c7bdf08c9cedca07fb80b004317e4451ecdea02304e: Status 404 returned error can't find the container with id 01bcbfafd092ad8fedeb9c7bdf08c9cedca07fb80b004317e4451ecdea02304e Nov 04 03:41:57.273516 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:57.273479 4784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:41:57.360602 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:57.360565 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" event={"ID":"449df0b7-c235-403b-bda3-991d5b85e355","Type":"ContainerStarted","Data":"01bcbfafd092ad8fedeb9c7bdf08c9cedca07fb80b004317e4451ecdea02304e"} Nov 04 03:41:59.374843 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:59.374808 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" event={"ID":"449df0b7-c235-403b-bda3-991d5b85e355","Type":"ContainerStarted","Data":"55f6f730baab3f9b4be9904c17eac4d43212064122a91c21b27756fded39e07f"} Nov 04 03:41:59.375401 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:59.374857 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" event={"ID":"449df0b7-c235-403b-bda3-991d5b85e355","Type":"ContainerStarted","Data":"318bd094c937780a1f28e20f4ee31ccd0fce6dc95a444321a0b036147d7bc76a"} Nov 04 03:41:59.495183 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:41:59.495127 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjk9m/must-gather-j8mkj" podStartSLOduration=2.331352581 podStartE2EDuration="3.49511308s" totalImagesPullingTime="1.163760499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:41:59.492585129 +0000 UTC m=+4166.459086121" watchObservedRunningTime="2026-11-04 03:41:59.49511308 +0000 UTC m=+4166.461614072" Nov 04 03:42:05.580012 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:05.579976 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-65l2h_95e3e816-aa1d-4022-b606-0b4ecc1ee79e/global-pull-secret-syncer/0.log" Nov 04 03:42:06.473535 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:42:06.473479 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:42:07.405226 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:07.403837 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8cz2p_8fe9ec5f-054e-4da3-b5d8-f2e4e32fd2ed/connection-checker/0.log" Nov 04 03:42:07.850570 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:07.850528 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8xkb7_64d03902-4d33-4bec-9725-53ec99ca7dbd/connection-checker/0.log" Nov 04 03:42:08.280323 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:08.280293 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bzwxn_6e9a6a6d-33fc-45ea-9b03-af9724367475/connection-checker/0.log" Nov 04 03:42:09.297113 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:09.297083 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qm2lj_b2376a8d-13b4-490b-9f66-b55cf5f3a51f/konnectivity-agent/0.log" Nov 04 03:42:10.542026 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:10.541995 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-209.us-west-1.compute.internal_41843a288696f78001f75198b2c4caf0/haproxy/0.log" Nov 04 03:42:15.544215 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:15.543556 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-94c54b89b-7zxv2_100c05c7-eb70-4e04-93a4-e405b1f2c0fa/authorino/0.log" Nov 04 03:42:16.299678 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:16.299643 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9vc9j_b280d45c-ddd2-4c1d-b0f0-d2aff81fba7a/manager/0.log" Nov 04 03:42:17.087764 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.087725 4784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv"] Nov 04 03:42:17.095348 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.095316 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.170719 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.170681 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-podres\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.170899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.170735 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/6d640d24-2be3-48ad-a698-d62590069ca7-kube-api-access-hjvbg\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.170899 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.170860 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-proc\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.171022 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.170982 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-lib-modules\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.171090 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.171028 4784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-sys\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.193166 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.193129 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv"] Nov 04 03:42:17.272338 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272301 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/6d640d24-2be3-48ad-a698-d62590069ca7-kube-api-access-hjvbg\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272360 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-proc\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272406 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-lib-modules\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272429 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-sys\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272500 4784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-podres\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272516 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-proc\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272563 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272549 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-lib-modules\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272556 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-sys\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.272887 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.272583 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d640d24-2be3-48ad-a698-d62590069ca7-podres\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.343500 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.343419 4784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvbg\" (UniqueName: \"kubernetes.io/projected/6d640d24-2be3-48ad-a698-d62590069ca7-kube-api-access-hjvbg\") pod \"perf-node-gather-daemonset-m7mrv\" (UID: \"6d640d24-2be3-48ad-a698-d62590069ca7\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.416830 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.416789 4784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:17.802690 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:17.802656 4784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv"] Nov 04 03:42:17.803900 ip-10-0-133-209 kubenswrapper[4784]: W1104 03:42:17.803859 4784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d640d24_2be3_48ad_a698_d62590069ca7.slice/crio-021db7f82f3d802dc35a5dba64cc3941ea31abef8a2a2fe5151896edde6c3b06 WatchSource:0}: Error finding container 021db7f82f3d802dc35a5dba64cc3941ea31abef8a2a2fe5151896edde6c3b06: Status 404 returned error can't find the container with id 021db7f82f3d802dc35a5dba64cc3941ea31abef8a2a2fe5151896edde6c3b06 Nov 04 03:42:18.282020 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:18.281990 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-68mzs_792ef3b5-5678-4de5-9764-86a3b17138c8/manager/0.log" Nov 04 03:42:18.531699 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:18.531654 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" event={"ID":"6d640d24-2be3-48ad-a698-d62590069ca7","Type":"ContainerStarted","Data":"bbd40a9923bfbaa5289023fd12ad42d94a2dc582de4ad57dafab4b0da4347afb"} Nov 04 03:42:18.531905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:18.531726 4784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" event={"ID":"6d640d24-2be3-48ad-a698-d62590069ca7","Type":"ContainerStarted","Data":"021db7f82f3d802dc35a5dba64cc3941ea31abef8a2a2fe5151896edde6c3b06"} Nov 04 03:42:18.531905 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:18.531753 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:18.650528 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:18.650427 4784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" podStartSLOduration=1.650413367 podStartE2EDuration="1.650413367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:42:18.649612588 +0000 UTC m=+4185.616113581" watchObservedRunningTime="2026-11-04 03:42:18.650413367 +0000 UTC m=+4185.616914360" Nov 04 03:42:23.904562 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:23.904528 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-tjjj9_c6c9ee7b-40be-4ecd-8459-bda33ae4fa2f/cluster-monitoring-operator/0.log" Nov 04 03:42:24.550134 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:24.550103 4784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-m7mrv" Nov 04 03:42:29.002281 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:29.002251 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj2n2_3aa62e5d-17b8-4174-8aba-200941de3f05/node-exporter/0.log" Nov 04 03:42:29.420989 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:29.420957 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj2n2_3aa62e5d-17b8-4174-8aba-200941de3f05/kube-rbac-proxy/0.log" Nov 04 03:42:29.841826 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:29.841735 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj2n2_3aa62e5d-17b8-4174-8aba-200941de3f05/init-textfile/0.log" Nov 04 03:42:34.468696 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:34.468652 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:42:34.474914 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:34.474888 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:42:34.556301 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:34.556268 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zp9vk_de579720-94f5-4cbd-b9f6-60e5b3d82647/console-operator/3.log" Nov 04 03:42:34.561968 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:34.561946 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82r7r_399e71a6-e15f-4aea-b7fc-25f45d6c182e/ovn-acl-logging/0.log" Nov 04 03:42:36.401291 ip-10-0-133-209 kubenswrapper[4784]: E1104 03:42:36.401245 4784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hl5s9" containerName="extractor" Nov 04 03:42:37.262959 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:37.262924 4784 scope.go:122] "RemoveContainer" containerID="4237ec42ac03c153f897333b7cd1e3a2128708649f36805843f5337292602d26" Nov 04 03:42:40.021211 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:40.021157 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b4848996xtj_f832f885-5e72-4892-8425-95fd377a536a/istio-proxy/0.log" Nov 04 03:42:41.559315 ip-10-0-133-209 kubenswrapper[4784]: I1104 03:42:41.559275 4784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-bdgtb_83ddc4ac-b046-42be-abf4-d5760f81e9d0/istio-proxy/0.log"